Cypress/CircleCi - Rerun workflow from failed causes 0 tests to run and 'passes' the circle job

[trevor-bennett]

Current behavior

splitting cypress/install and cypress/run into two different jobs within the circleci config yml will cause 0 tests to actually run when selecting rerun workflow from failed if the failed job within circle is only cypress/run

First failed circleci run:

After selecting Rerun workflow from failed:

Cypress console logging:

Desired behavior

When the CircleCI option to Rerun workflow from failed step in invoked, the proper tests are ran.

Test code to reproduce

Prerequisites:

• cypress dashboard (record = true)
• circleci/cypress orb

CircleCi Job Definitions:

version: 2.1

orbs:
  cypress: cypress-io/cypress@2.1.0

jobs:
  - cypress/install:
      requires:
        - install
      post-steps:
        - run:
            name: Print machine info
            command: npx cypress info
  - cypress/run:
      start: npm run serve
      wait-on: "http-get://localhost:8000"
      record: true
      parallel: true
      parallelism: 2
      ci-build-id: ${CIRCLE_SHA1:0:8}
      group: "UI Tests - Electron"
      requires:
        - cypress/install
      post-steps:
        - report-cypress-results

Cypress Version

10.3.1

Node version

12.19.0

Operating System

Ubuntu 20.04.4 LTS

Debug Logs

No response

Other

No response

[admah]

@trevor-bennett thanks for taking the time to report this. I believe it is a duplicate of this issue in our github-action repo. Could you retry using 10.7.0 or later to see if you still run into the issue?

[trevor-bennett]

@admah I updated to 10.9.0 and the issue still occurs. The rerun from failed is causing the same logging/situation where 0 tests are run but is listed as 27+ specs found and passes the entire step

[admah]

@trevor-bennett I'm going to transfer this issue to our circleci-orb repo so we can triage it appropriately there since this is more than likely an issue with that Orb and "Rerun workflow from failed".

[trevor-bennett]

Turns out to be any re-run job is skipping every single test regardless if its from start or failed when run against cypress 10.9.0 and the cypress: cypress-io/cypress@2.1.0 orb

[jnpwebdeveloper]

Experiencing the same issue here with cypress-io/cypress@2.2.0 and cypress 12.2.0.

[jnpwebdeveloper]

For anyone else experiencing this issue, we solved it by appending the build number to the commit hash identifier in the ci-build-id attribute. This means when we re-run this, the value is unique and therefore does a full re-run as you would expect.

    - cypress/run:
        ...
        ci-build-id: ${CIRCLE_SHA1:0:8}-${CIRCLE_BUILD_NUM}

Hope this helps anyone else out there.

[jennifer-shehane]

As stated by @jnpwebdeveloper - you want to ensure that there is a unique ci-build-id for each run and each rerun. Typically Cypress is pretty good about determining a ci-build-id for reruns, but every CI environment is different and if you've overwritten this to be static for each run, that could also cause the issue where on rerun, Cypress sees the rerun as a previous run that's already finished and exists (unfortunately with passing tests).

For v 3.0.0 of our orb, you could pass --ci-build-id to the cypress-command : cypress-command: 'npx cypress run -ci-build-id=${CIRCLE_SHA1:0:8}-${CIRCLE_BUILD_NUM}'

Closing as the solution is outlined above.