No retry attempts are logged in stdout when setting different reporter other than `spec`

[gustaferiksson]

Current behavior:

cypress run doesn't output retries logging in the terminal when using reporters other than spec.

Desired behavior:

Having cypress run to always output retries logging in the terminal (at least be able to enable it).

This was possible in the cypress retries plugin with this:

module.exports = (on, config) => {
  require('cypress-plugin-retries/lib/plugin')(on)
}

Test code to reproduce

run a failing test with any mocha reporter except spec e.g cypress run --reporter list

Versions

Cypress 5.0.0

[jennifer-shehane]

I can recreate this. I do see that the attempts are not logged within the test when running with the list reporter.

4.12.1

CYPRESS_RETRIES=3 cypress run --reporter list

support/index.js

require('cypress-plugin-retries')

plugins/index.js

module.exports = (on, config) => {
  require('cypress-plugin-retries/lib/plugin')(on)
}

spec.js

it('test', () => { 
  expect(true).to.be.false
})

package.json

{
  "devDependencies": {
    "cypress": "4.12.1",
    "cypress-plugin-retries": "^1.5.2"
  }
}

5.0.0

cypress run --reporter list

spec.js

it('test', {
  retries: 3
}, () => { 
  expect(true).to.be.false
})

package.json

{
  "devDependencies": {
    "cypress": "5.0.0"
  }
}

You can see that the test retried 3 times because it took 4 attempt screenshots, but it's not logged in the reporter output.

[kuceb]

Yes, in the plugin it did print the retry logs, however that just a hack. If you opt into a custom reporter you're telling Cypress you want to control the reporting - so we don't print anything. The mocha list reporter does not print retries info

As an example if you provided a custom xml reporter, it wouldn't make sense for us to log retries to the console.

Your custom reporter will need to listen to the retry event if you want to implement retries logging similar to the default reporter.

[gustaferiksson]

@Bkucera I'll kindly have to disagree due to the fact that spec also is a mocha reporter that doesn't have retries logging by default and yet cypress prints retires when I opt into a 'custom' reporter by running with --reporter spec. Therefor it would make more sense to be able to enable retries logging with some property in the cypress configuration file.

[kuceb]

right, It's a reasonable request for us to add retry info to existing internal mocha reporters. Reopening

[jennifer-shehane]

Many people are running into this issue when defining the mocha-multi-reporters as their reporter, which I suppose is outside of the scope that Ben originally defined, but this is a pretty popular reporter. Any way to accommodate printing retries in that reporter also?

[lissetb]

I've run into the same issue while using the cypress-testrail-reporter. Confirmed that only when remove reporter option will the retries work.

[Hazzard17h]

As a workaround you can use:

// plugins/index.js

module.exports = (on, config) => {
  on('task', {
    log: param => console.log(...Array.isArray(param) ? param : [ param ]) || true
  });
};

// support/index.ts

const config: any = Cypress.config();
if (!config.isInteractive && config.reporter !== 'spec') {
  afterEach(() => {
    const test = (cy as any).state('runnable')?.ctx?.currentTest;
    if (test?.state === 'failed' && test?._currentRetry < test?._retries) {
      cy.task('log', `    (Attempt ${test._currentRetry + 1} of ${test._retries + 1}) ${test.title}`, { log: false });
    }
  });
}

[laerteneto]

This same issue is happening to me in CI mode when using another reporter:

  "reporter": "mochawesome",
  "reporterOptions": {
    "reportDir": "cypress/test-results/",
    "mochaFile": "cypress/test-results/my-test-output.xml",
    "jenkinsMode": false,
    "quite": true,
    "overwrite": false,
    "html": false,
    "json": true
  },

When I remove the reporter, the retries works fine.

I am using Cypress 7.3.0

[Kingy14]

I have the same issue as @laerteneto on Cypress 9.1.1

Thanks @Hazzard17h for the workaround, this worked great for me.

[GarrisonD]

The same issue is on Cypress 10.3.1 😢

[rezasadeghi1]

Still having same issue on 13.0.1. This need to be solved.

[Marko-Ichev]

Still having same issue on 13.0.1. This need to be solved. You can use @Hazzard17h workaround until they fix it. @rezasadeghi1

[geritol]

Experiencing this issue on 13.2 with reporter mochawesome-reporter.

Workaround of @Hazzard17h works fine, did anyone figure out how to also log the failure cause of the specific retry?

[briangleeson]

Workaround of @Hazzard17h works fine, did anyone figure out how to also log the failure cause of the specific retry?

Iterating on @Hazzard17h's suggestion, using this to include the failure of the retry in the logs:

if (!config.isInteractive) {
  beforeEach(() => {
    const test = cy.state("runnable")?.ctx?.currentTest;
    if (test?._retries > 0) {
      // When retries are enabled, always log the attempt number
      cy.task("log", `==> Attempt ${test._currentRetry + 1}, max attempts = ${test?._retries + 1}: ${test.title}`, { log: false });
    }
  });
  afterEach(() => {
    const test = cy.state("runnable")?.ctx?.currentTest;
    if (test?.state === "failed" && test?._currentRetry < test?._retries) {
      // When an attempt fails and isnt the final attempt, log out the error stack
      cy.task("log", `    Error in attempt ${test._currentRetry + 1}: ${test.err.stack}`);
    }
  });
}

Output:

==> Attempt 1, max attempts = 2: MySimpleTest
    Error in attempt 1: AssertionError: Timed out retrying after 5000ms: Expected <h1.title-text> not to exist in the DOM, but it was continuously found.
    at Context.eval (webpack:///../tests/mySimpleTest.spec.js:34:44)
==> Attempt 2, max attempts = 2: MySimpleTest
...

[Alecsou]

Still a problem with Cypress 13.13.0, Node 18.19.0, and junit and mocha-junit-reporter

DevTools listening on ws://127.0.0.1:54915/devtools/browser/b37fb9a8-97d0-4016-97ab-b4b0d86eec85

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.13.0                                                                        │
  │ Browser:        Edge 126 (headless)                                                            │
  │ Node Version:   v18.19.0 (C:\Program Files\nodejs\node.exe)                                    │
  │ Specs:          2 found (CrossingTheMFA.cy.js, ************************.cy.js)                 │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  CrossingTheMFA.cy.js                                                            (1 of 2)
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="18.773" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2024-07-15T10:21:24" tests="0" file="cypress\e2e\CrossingTheMFA.cy.js" time="0.000" failures="0">
  </testsuite>
  <testsuite name="Crossing the MFA" timestamp="2024-07-15T10:21:24" tests="1" time="18.766" failures="1">
    <testcase name="Crossing the MFA tests Crossing the MFA" time="3.918" classname="tests Crossing the MFA">
      <failure message="Unspecified AssertionError" type="AssertionError"><![CDATA[AssertionError: Unspecified AssertionError
    at Context.eval (webpack:///./cypress/e2e/CrossingTheMFA.cy.js:13:1)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  3                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     18 seconds                                                                       │
  │ Spec Ran:     CrossingTheMFA.cy.js                                                             │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  (Screenshots)

  -  C:\Users\*******\Desktop\Dev\WebUITesting\cypress\screenshots\CrossingTheMFA.cy.     (1256x629)
     js\Crossing the MFA -- tests Crossing the MFA (failed).png
  -  C:\Users\*******\Desktop\Dev\WebUITesting\cypress\screenshots\CrossingTheMFA.cy.     (1256x629)
     js\Crossing the MFA -- tests Crossing the MFA (failed) (attempt 2).png
  -  C:\Users\*******\Desktop\Dev\WebUITesting\cypress\screenshots\CrossingTheMFA.cy.     (1256x629)
     js\Crossing the MFA -- tests Crossing the MFA (failed) (attempt 3).png

[mistercrunch]

IMNSHO default behavior should be to log the retires. Come on!

[mistercrunch]

From what I see, when the headless browser crashes, the retries don't seem to work. Seems it would be the whole point of a retry - retries are especially relevant/useful when the browser crashes.