Option to abort on first failure

[KittyGiraudel]

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

• Cypress Version: 0.19.2

[KittyGiraudel]

In the mean time, this does the trick:

Cypress.on('fail', () => Cypress.runner.abort())

[brian-mann]

In 0.20.0 this became:

Cypress.on('fail', () => {
  Cypress.stop()
})

[KittyGiraudel]

From what we’ve seen, Cypress.stop() exits the process with a 0 code, which means even if a test fails, this will make it behave like everything went fine. We believe Cypress.runner.abort() did exit with a non-0 code in case of a failure.

Are we mistaken?

[brian-mann]

I didn't try it, but Cypress.stop() does do more things than aborting just the runner.

Try this:

Cypress.runner.stop()

That's where the abort code moved to, but even this does a bit more than it used it. May work though.

[tomaswitek]

Any update here? Both Cypress.runner.stop() and Cypress.stop() behave like everything went fine.

[brian-mann]

Nope. PR's are welcome.

[ThomasDecavele]

This is a much desired feature!

In the meanwhile I managed to get the workaround working though, here's what's in my "support/index.js":

import './commands'
import { OperationCanceledException } from 'typescript';

Cypress.on('fail', () => {
    Cypress.stop();
    throw new OperationCanceledException();
});

[nanoflat]

I think it will work in support/index.js, not in plugins/index.js

[01taylop]

The use-case I have for this is that I need my authentication tests to pass first - if they don't all subsequent tests will fail. Although the other tests will fail in the before hooks, it still slows things down unnecessarily (30 seconds per test file).

I have to name my test 1auth so that it runs first and whitelist a cookie which contains a token required for the other tests to pass.

Cypress.Cookies.defaults({
  whitelist: 'token',
})

I was hoping to do something like this in my auth tests to stop running tests on fail;

  before(() => {
    Cypress.on('fail', () => {
      Cypress.runner.stop()
    })
  })

It does stop the test runner, but it doesn't fail any tests so I get 3/3 tests passed and a green build which is not the intended outcome. I also tried adding am intentional failing test before stopping cypress, but then the test runner continues.

[danh91]

I have a temporary solution that I think can help someone.

I just add an after each in my utilities file imported by all my specs.

afterEach(function() {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

I take advantage of Mocha offering a way to check the state of the last test. And the nice part is that the process exit with a failing code (not 0)

[sanderdejong88]

@DanH91 I've been following this thread for months, and finally someone found a way! Awesome! I've put your code in the support file which also seems to work.

[harvitronix]

This no longer works for me with Cypress 3.x.

[jennifer-shehane]

@harvitronix We changed how specs are run in 3.0 so that the files are run isolated. This means the before runs before each spec file - so I am guessing that it is stopping and failing that 1 spec file and then continuing to the next file. Is this what is happening?

[brian-mann]

Yes we've waited to implement this feature due to how aborting is much more complex now.

We likely need to allow for multiple abort strategies

abortStrategy: 'spec' | 'run'

One would abort just the spec, one would abort the entire run.

This becomes even more complex with parallelization.

[harvitronix]

@jennifer-shehane that's exactly right. Totally understand it gets much more complex now. Would love to have the options you describe, @brian-mann.

[vedmant]

Is there a way to stop execution but run after() method?

[mattvb91]

:+1:

[mohsenny]

I tried your solutions and the runner correctly stops. However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s. Any suggestion?

An example: As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

[drumheadrob]

Hi, loving Cypress, but not being able to stop our CI build for a failing test is stopping me from being able to use this in CI, and will especially be a problem in CD.

I've searched quite thoroughly and hope I'm missing something somewhere, but it seems like this is related to this thread?

If so, is there any progress / planned work on this solution? Cheers.

[bahmutov]

@drumheadrob your issue is very different - you are saying that the cypress run does NOT exit with code 1 after all tests complete and some fail?!

[jennifer-shehane]

We are currently working on the underlying processes needed to 'cancel' a run and any other queued spec files, which is a precursor to the work needed to 'abort' the rest of the tests on first failure.

[pacop]

Some news right here? I am facing the very same problem, we were not able to use cypress on our CI.

[stclairdaniel]

I'm also interested in this.

[nantoniazzi]

Are you still working on this feature ? The more we add tests with Cypress, the more it becomes unusable in our CI

[cmcnicholas]

We have the same situation, paying for cypress dashboard on private repos but our tests (now ~400 or so) are taking more and more time to complete. Much needed to stop early so our pipeline is freed up for the next PR to execute.

[vladarefiev]

It will be cool feature

[gurre]

In order to fail the test make sure to throw the error.

// cypress/support/index.js
Cypress.on('fail', error => {
  Cypress.runner.stop()
  throw error; // throw error to have test fail
});

[itslenny]

@gurre is this working for you?

I'm doing this in my main describe block. The test run is stopping, but it's exiting as if it was successful.

  before(() => {
    Cypress.on('fail', error => {
      Cypress.runner.stop()
      throw error;
    });
  });

edit: nvm... I totally missed @DanH91 's answer above ( https://github.com/cypress-io/cypress/issues/518#issuecomment-373369129 ) which works perfectly with no need to explicitly throw.

[TrueMan777]

I wanted to do the fail-fast-all type of approach so I've ended up adding the following to my cypress/support/index.js file. Hopefully, it helps someone else.

// Fail-fast-all-files
before(function () {
    cy.getCookie('has-failed-test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

// Fail-fast-single-file
afterEach(function () {
    if (this.currentTest.state === 'failed') {
        cy.setCookie('has-failed-test', 'true');
        Cypress.runner.stop();
    }
});

[vesper8]

@TrueMan777

Hi and thanks for sharing your solution! I'm rather new to Cypress. I tried dumping the above code inside my cypress/support/index.js but that is throwing an unexpected error. Do you need to wrap it or do something extra?

This is my index.js before and after

// Import commands.js using ES2015 syntax:
import './commands';

// Import commands.js using ES2015 syntax:
import './commands';

// Fail-fast-all-files
before(function () {
    cy.getCookie('has-failed-test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

// Fail-fast-single-file
afterEach(function () {
    if (this.currentTest.state === 'failed') {
        cy.setCookie('has-failed-test', 'true');
        Cypress.runner.stop();
    }
});

I am now receiving this error:

/node_modules/@babel/runtime-corejs2/helpers/esm/typeof.js:1
import _Symbol$iterator from "../../core-js/symbol/iterator";
^
ParseError: 'import' and 'export' may appear only with 'sourceType: module'

This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

- A missing file or dependency
- A syntax error in the file or one of its dependencies

Fix the error in your code and re-run your tests.

Any idea what's wrong? Thanks!!

[TrueMan777]

A wild guess, could change the before clause to :

before(function () {
    cy.getCookie('has-failed-test').then(function(cookie) {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

[vesper8]

I just removed the typeof cookie === 'object' since that was causing problem and I don't think it's absolutely necessary

My specs only contain a single test, I want all specs to be aborted on first fail, and this isn't causing this to happen for me.. is it working like that for you?

[TrueMan777]

I just removed the typeof cookie === 'object' since that was causing problem and I don't think it's absolutely necessary

My specs only contain a single test, I want all specs to be aborted on first fail, and this isn't causing this to happen for me.. is it working like that for you? Interesting, I would never think that condition could trigger ParseError: 'import' and 'export' may appear only with 'sourceType: module' error.

@vesper8, it won't fail-fast the following tests but skip them. So you will still have the tests in the table but the execution time is now 1-2s. This works fine with me, at least for now. It is a workaround anyway.

[pawelrudnicki]

Did you find anything that would work with 3.3.1 version?

[bellsenawat]

I try to using this script and Cypress not stop when fist test failed :( But It skipped and continue test another spec

[oskargustafsson]

The following works for me in Cypress 3.3.2 (solution based on prev. comments in this issue)

// in cypress/support/index.js

switch (Cypress.env('abort_strategy')) {
  case 'run':
    // eslint-disable-next-line no-undef
    before(function onBeforeEach() {
      // Skips any subsequent specs, if the run has been flagged as failed
      cy.getCookie('has_failed_test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
          Cypress.runner.stop();
        }
      });
    });
  /* fallthrough */
  case 'spec':
    afterEach(function onAfterEach() {
      // Skips all subsequent tests in a spec, and flags the whole run as failed
      if (this.currentTest.state === 'failed') {
        cy.setCookie('has_failed_test', 'true');
        Cypress.runner.stop();
      }
    });
    Cypress.Cookies.defaults({
      whitelist: 'has_failed_test',
    });
    break;
  default:
}

This lets you control the abort behavior using the cypress_abort_strategy env var, e.g. export cypress_abort_strategy=spec && cypress run.

Would be nice not to have to hack it like this though!

[Undistraction]

Same use-case as a lot of others here. We now have a large-enough test-suite that it is becoming impractical to let the CI run on past a failing test. We need to way of aborting on the first failure.

[vedmant]

Is there a way to stop tests after first fail but still run after() hook? I use to to clean temp data that created for test.

[ericorruption]

Bummer that this is open for over a year with no official solution. :/ None of the workarounds suggested worked for me.

[philjones88]

I too want this, when cypress flake happens I want to be able to stop the run.

The problem is I use a pool of machines so the workarounds all seem targeted at just 1 cypress machine?

[ericorruption]

Not even in this case. I don't use parallel runs and it still doesn't work.

On Thu, Jul 25, 2019 at 12:24 PM Phil Jones notifications@github.com wrote:

I too want this, when cypress flake happens I want to be able to stop the run.

The problem is I use a pool of machines so the workarounds all seem targeted at just 1 cypress machine?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/518?email_source=notifications&email_token=AASDX7QBEIYC7V6AYO2L5WTQBF5MDA5CNFSM4DNFROL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2ZBZAI#issuecomment-514989185, or mute the thread https://github.com/notifications/unsubscribe-auth/AASDX7TLECVHK4OZC3Z3SPLQBF5MDANCNFSM4DNFROLQ .

-- Eric Quanz Front-end focused developer ericquanz.com / +49 160 9571 0135

[vesper8]

unbelievable it's been 2 years and a half and such a simple thing that others have long included is still missing?? there's seriously still no built-in way to run ./node_modules/cypress/bin/cypress run and abort on first failure??

[vaske]

what about in cypress.json:

{
  "maxFailedTests": 1
}

[AnshuChaudhari]

I tried your solutions and the runner correctly stops. However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s. Any suggestion?

An example: As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

@mohsenny I am searching a solution for exactly this same issue, where the rest of the test cases dint execute if one of the iteration fails. Did you found any solution for this ?

[ksmith-karma-police]

also interested in this feature (stop current test scenario/file)!

[jpike88]

Yeah this is a major pain. It would be nice if the cypress team at least had an officially recommended workaround inside of people shooting in the dark.

This snippet didn't work.

afterEach(function () {
    if (this.currentTest && this.currentTest.state === 'failed') {
        (Cypress as any).runner.stop();
    }
});

[dwelle]

I've modified one of the recommendations here and posted it as an answer on SO a week back: https://stackoverflow.com/a/58660504/927631

Here's the code:

// cypress/plugins/index.js

let shouldSkip = false;
module.exports = ( on ) => {
  on('task', {
    resetShouldSkipFlag () {
      shouldSkip = false;
      return null;
    },
    shouldSkip ( value ) {
      if ( value != null ) shouldSkip = value;
      return shouldSkip;
    }
  });
}

// cypress/support/index.js

function abortEarly () {
  if ( this.currentTest.state === 'failed' ) {
    return cy.task('shouldSkip', true);
  }
  cy.task('shouldSkip').then( value => {
    if ( value ) this.skip();
  });
}

beforeEach(abortEarly);
afterEach(abortEarly);

before(() => {
  if ( Cypress.browser.isHeaded ) {
    // Reset the shouldSkip flag at the start of a run, so that it 
    //  doesn't carry over into subsequent runs.
    // Do this only for headed runs because in headless runs,
    //  the `before` hook is executed for each spec file.
    cy.task('resetShouldSkipFlag');
  }
});

Will skip all further tests once a failure is encountered. The output will look like:

[jmfn]

@dwelle thanks for this.

I've found this works well w/ cypress run invocations and CI use cases, but not when using the GUI runner. Appears the shouldSkip state scoped at the plugin module is not reset between runs. Haven't determined a way yet to reset it without exiting and reopening the runner.

[dwelle]

My bad --- haven't tested that scenario. I've updated the answer. It's an easy fix --- reset the flag on each run (must be done only for headed runs. See the comments in the code, for more).

[seyfer]

@dwelle would be great to stop it, not showing any output in the end. Now it goes through every file after the first fail and spends 1s for each, just to mark it skipped. Given I have 60 files - I need to wait 1 minute just to get failed output at the end.