Cypress parallelization and BitBucket Pipelines rerun failed steps

[wwei-flux]

What would you like?

We currently use BitBucket Pipelines to run Cypress tests in parallel. Same setup as introduced here https://docs.cypress.io/guides/continuous-integration/bitbucket-pipelines

BitBucket Pipelines support to only rerun failed steps https://bitbucket.org/blog/rerun-failed-pipeline-steps It can be quite useful.

We found when rerun a few failed steps that use Cypress parallelization, they will run all specs again, not just specs assigned to the failed steps.

For example, if I have 5 specs and 5 parallel steps, each spec is assigned to one step on the first run. Step 1 to step 4 succeeded and step 5 failed. If I trigger a "rerun failed step" to only rerun step 5, step 5 will run all 5 specs again. I expect step 5 to only run spec 5 in this case.

Why is this needed?

This is a useful function when tests are affected by external dependencies (e.g. a docker image that is pulled into test environment that has issues, or a BitBucket env var config has issues).

Ideally, Cypress Dashboard should remember the mapping between specs and parallel steps, and when a "rerun failed step" happens, only rerun the originally assigned specs.

Other

No response

[brentatkins]

I've run into a similar issue, except on Azure DevOps. Would be great to have the ability to rerun a subset of tests based on the parallelization split.

[jjwong]

Seeing the same issue in our bitbucket pipelines.

[mystic-poet]

any workaround for this issue?

[carlos-nunez]

Seeing the same issue on bitbucket pipelines. It would be really great to be able to rerun only the failed test. It wastes a ton of time.

[GrayedFox]

Posting the below workaround for future eyeballs. It's not ideal but as workarounds go, it gets our team pretty close to what we need. We save the names of failed test files using the cy after:spec hook and then take advantage of passing artifacts between build steps in the pipeline.

Due to BitBucket limitations, the only way this is possible is to fudge the exit code of the initial parallel test run attempt to always be 0.

Cypress events hooks:

// cypress.config.js

const { writeFile, readdir, unlink } = require('node:fs/promises');
const { readFileSync } = require('fs');
const { defineConfig } = require('cypress');

const cypressEnv = JSON.parse(readFileSync('./cypress.env.json', 'utf-8'));

module.exports = defineConfig({
    e2e: {
        setupNodeEvents(on, config) {
            // before each run, delete contents of temp directory if cleanup set to true
            on('before:run', async () => {
                if (!cypressEnv.cleanup_failed_tests) {
                    return;
                }
                try {
                    const dirName = 'cypress/temp';
                    for (const file of await readdir(dirName)) {
                        if (file === '.gitkeep') continue;
                        const fullPath = `${dirName}/${file}`;
                        console.log('Removing previously failed spec ref: ' + fullPath);
                        await unlink(fullPath);
                    }
                } catch (err) {
                    // log the error but don't prevent test runs if we fail to remove directory contents
                    // eslint-disable-next-line no-console
                    console.error(err);
                }
            });

            // after each spec, write the name of the spec file, replacing forward slashes with hyphens
            on('after:spec', async (spec, results) => {
                if (results) {
                    // check for a final failed state - this means tests that are retried
                    // but then pass won't be repeated
                    const failures = results.tests.some(test =>
                        test.attempts.some(attempt => attempt.state === 'failed')
                    );

                    if (failures) {
                        try {
                            await writeFile(
                                `cypress/temp/${spec.relative.replaceAll('/', '-')}`,
                                spec.relative,
                                'utf-8'
                            );
                        } catch (err) {
                            // log the error but don't fail test runs if we fail to write the file ref
                            // eslint-disable-next-line no-console
                            console.error(err);
                        }
                    }
                }
            });
        },
        specPattern: 'cypress/e2e/**/*.spec.{js,jsx,ts,tsx}'
    }
});

Sample BitBucket pipeline using parallel workers and then retrying failed tests in the next step:

---
image: node:lts

script-anchors:
    # build your app
    build-it: &build-script |
        npm ci
        echo "whatever else you need to do"

    # Conditionally run e2e tests
    e2e-tests: &e2e-tests-script |
        npm run cypress -- --record --parallel --group CI-Electron --ci-build-id "$BITBUCKET_BUILD_NUMBER" || true

    # Read failed tests read from cypress/temp directory and run again
    e2e-failed-tests: &e2e-failed-tests-script |
        # list all contents of temp directory and replace hyphens with slashes and newlines with commas
        BITBUCKET_CYPRESS_TESTS=$(ls cypress/temp/ | tr '-' '/' | tr '[:space:]' ',' | sed 's/.$//')
        echo "Failed Cypress test string is: $BITBUCKET_CYPRESS_TESTS"

        if [[ -z $BITBUCKET_CYPRESS_TESTS ]]; then
            echo "NO TEST FAILURES! WOOT!"
        else
            echo "RUNNING E2E FAILED TESTS"

            npm ci
            npm run cypress -- \
            --record \
            --group CI-Electron-Retry \
            --ci-build-id "$BITBUCKET_BUILD_NUMBER-Retry" \
            --spec "$BITBUCKET_CYPRESS_TESTS"
        fi

definitions:
    # Test artifacts, make sure cypress/temp exists and is writable
    test-artifacts: &test-artifacts
        - cypress/screenshots/**
        - cypress/videos/**
        - cypress/temp/**

    # Common steps
    steps:        
        - step: &build
            name: Build
            script:
                - *build-script

        - step: &e2e-tests
            <<: *base
            name: Parallel E2E Tests (All)
            artifacts: *test-artifacts
            script:
                - *e2e-tests-script

        - step: &e2e-failed-tests
            <<: *e2e-tests
            name: Failed E2E Tests (Retry)
            script:
                - *e2e-failed-tests-script

pipelines:
    branches:
        'testing/**':
            - step:
                <<: *build
                name: Build It
            - parallel:
                fail-fast: false
                steps:
                    # Run these 8 steps in parallel
                    [ { step: *e2e-tests }, { step: *e2e-tests }, { step: *e2e-tests }, { step: *e2e-tests },
                        { step: *e2e-tests }, { step: *e2e-tests }, { step: *e2e-tests }, { step: *e2e-tests } ]
            - step: *e2e-failed-tests
...

This doesn't workaround the core issue (that is: re-running the failed parallel workers will still behave whatever way the Cypress cloud tell them to) but assuming a healthy test base with minimal flake, you're looking at a repeatable "failed tests only" step, which I think is what most people want when they land here.

Note that fail-fast is set to false. This is important, otherwise the first test failure will halt all other workers and they might no be done running all your tests, so you would potentially end up with a fully green pipeline that hasn't run all of your tests, as always, be careful when copy-pastying strange code from the interwebs.