Open proCrichard opened 5 years ago
@proCrichard could you mention what plugin you are using?
I wrote a custom plugin to send test results to datadog.
Starts with an after
hook:
after(() => {
const findTests = suite => {
suite.tests.forEach(test => {
cy.task('logTestResults', {
title: `${test.parent.title}${test.title}`,
state: test.state,
});
});
if (suite.suites.length < 1) {
return;
}
suite.suites.forEach(nestedSuite => {
findTests(nestedSuite);
});
};
cy.state('runnable').parent.suites.forEach(suite => {
findTests(suite);
});
});
Which triggers this event listener:
const dogapi = require('dogapi');
const changeCase = require('change-case');
module.exports = (on, app) => {
const dogapiOptions = {
api_key: process.env.DATADOG_API_KEY,
app_key: process.env.DATADOG_APP_KEY,
};
dogapi.initialize(dogapiOptions);
on('task', {
logTestResults({ title, state }) {
const tags = [];
//This if statement is to catch state=undefined which is sent from tests that failed and were retried.
if (!state) {
state = 'failed';
}
tags.push(`outcome:${state}`);
tags.push(`description:${changeCase.snakeCase(title)}`);
tags.push(`app:${app}`);
dogapi.metric.send(`app.e2e`, 1, { tags }, (error, results) => {
console.dir(results); // eslint-disable-line no-console
});
return null;
},
});
};
did your test that failed (but had state: passed) fail from a beforeEach hook or in the test body?
What retry number do you have enabled?
Retries set to 2 I had the same outcome when I forced a test to fail in:
before()
hookbeforeEach()
@proCrichard alright, and what version of cypress and this plugin?
I just added your after() to my spec for this project, and I am getting the correct state set on the runnables
"cypress": "3.4.0",
"cypress-plugin-retries": "1.2.2"
@proCrichard ok, I think I am reproducing it now. Thanks
Ok great. I appreciate you jumping on this for me.
@proCrichard after spending a couple hours on this, I'll need to edit the Cypress source directly to fix this, but I have a PR to do so here https://github.com/cypress-io/cypress/pull/3968
I would recommend using the module API to pull results out instead of using runnable instances. https://docs.cypress.io/guides/guides/module-api.html#cypress-run. However, I know that can be cumbersome
When a test fails and is retried, that test's details in the
cy.state('runnable')
object is somehow broken. From an example I have here, I can see in the cypress logger that my test has failed:However the
cy.state('runnable')
output on that test considers it 'passed':This is throwing off a plugin I use to report test results. Any information you have on this issue would be helpful.