Option to abort on first failure

KittyGiraudel commented 7 years ago

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

Cypress Version: 0.19.2

Ciniok commented 4 years ago

Is there any workaround for cypress 3.8 ? I don't have much time for coming up with my own and it's super imporant to be able to fail fast when in the CI

Mumcio commented 4 years ago

Any update here?

Arver1 commented 4 years ago

Hi, cypress 4.2.0. That is quick and easy solution:

afterEach(function () {
        if (this.currentTest.state === 'failed') {
            throw Error();
        }
    });

https://www.npmjs.com/package/cy-webpack-plugin or you can write me and i can write node.js script if you do not use webpack.

Mumcio commented 4 years ago

@Arver1 yes but in this way, by throwing an error you will lose the reason why test failed

dialex commented 4 years ago

FYI this code snippet no longer works with the latest Cypress version, when running tests visually. Documented issue at https://github.com/cypress-io/cypress/issues/6414#issuecomment-612933329

(...) For that mechanism to work, the cookie needs to be whitelisted, so that it persists between tests of the same test run. After we get one failure, the test suite is aborted. When we try to rerun it, the cookie is still there (whitelisted), the before sees its latest value (true) and immediately aborts the test run without running any test.

biiiipy commented 4 years ago

I can't imagine how can anyone use cypress without the ability to stop on error. It's insane that this issue is open for 3 years

Mumcio commented 4 years ago

Indeed, I fully agree with you @biiiipy. I can not believe it is not resolved so long too. In addition, the lack of possibility to stop tests on fail will extend pricing plans on any CI.

It is elementary. Basic. Do you have this on your list @jennifer-shehane ?

PierreCavalet commented 4 years ago

As a workaround we use a script and manually list all the test we want to use:

  cypress run --spec="cypress/integration/test1.js" &&
  cypress run --spec="cypress/integration/test2.js" &&
  cypress run --spec="cypress/integration/test3.js" 
  exit $? # in case this is not the end of your script because you are in a "if" or something

It's not perfect but it works so you can use this as a workaround. I did not evaluate the cost of relaunching cypress everytime compared to the execution time you loose when it fails but if anyone wants to provide feedbacks, you are more than welcome.

Hopefully we will have a solution for this by then.

CodeTroopers commented 4 years ago

Could we have a status on this feature by the cypress team? After 3 years and the criticality of the need, it will be nice to have a view on the next. @jennifer-shehane

biiiipy commented 4 years ago

The weird thing is that this "feature" is 1) super essential 2) should be quite easy to implement in the right place and 3) it hasn't been implemented for 3 years (especially, imo, it should have been the default behavior of the test runner since the beginning).

drumslave-git commented 4 years ago

I used such hack for now

describe(`Some title" app`, function () {
    let failedTest;
    beforeEach(function () {
        cy.wrap(failedTest).should('be.undefined')
    });
    afterEach(function () {
        if (this.currentTest.state === 'failed') {
            failedTest = this.currentTest.title;
        }
    });
})

So as soon as one of tests will fail - beforeEach check also will be failed.

sesam commented 4 years ago

@biiiipy I'm doing like this:

Cypress.on("fail", (error, runnable) => {
  if (runnable.type === "test") {
    // (here we do some statistics on which specific step failed)

    runnable.parent._bail = true; // WORKS. Now, to find the officially sanctioned way to do this... :)
  }
  throw error; // throw error to have test still fail
});

asandoval1 commented 4 years ago

I tried your solutions and the runner correctly stops. However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s. Any suggestion? An example: As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

@mohsenny I am searching a solution for exactly this same issue, where the rest of the test cases dint execute if one of the iteration fails. Did you found any solution for this ?

@mohsenny , @AnshuChaudhari I'm looking for a solution to this issue as well. Have you found anything useful so far?

asandoval1 commented 4 years ago

@jennifer-shehane, @bahmutov Are there any updates on this one? This is a huge inconvenience that's preventing us from moving towards a nice and smooth CI integration for testing modules with multiple specs.

boredland commented 4 years ago

hi, is there a workaround that works with cypress 4.7.0? when I do

if (this.currentTest.state === 'failed') {
  throw Error();
}

this only seems to stop the current spec, not the cypress run.

mduft commented 4 years ago

Would also require a solution expecially for CIs - our suite runs ~30 minutes, and most failures occur in the first 5 minutes... Thats just a massive waste of resources and time.

jakedowns commented 4 years ago

~~same here. been using cypress for a week and a half now and it's one "wont-fix" after another. i really love the concept, and the tooling provided, but these limitations are very frustrating.~~

:bow:

jennifer-shehane commented 4 years ago

From https://github.com/cypress-io/cypress/issues/1599 - expressed a want to set a maxFailures option instead of exit on exact first, so could maybe make this configurable.

@jakedowns This feature is not a 'wont-fix', it is labeled as a 'proposal', which means no work has been done on this issue as of today but it is still a consideration for future feature work. We do not have an estimate on when this will be delivered.

drumslave-git commented 4 years ago

@jakedowns also please remember that cypress is open source and very flexible, means we are able to achieve any required behavior in theory. My hack that Iv posted here works pretty fine for me.

jennifer-shehane commented 4 years ago

Went through the thread of comments to consolidate some workarounds. I don't think any of them completely meets the requirements that everyone wants for this feature, but you would be able to partially abort on first failure.

Tests that are skipped or pending do not count as test recordings for billing.

abort using this.skip

This workaround is probably the most comprehensive. Use this.skip to properly skip any remaining tests run after the failed test. Thanks @dwelle https://github.com/cypress-io/cypress/issues/518#issuecomment-552382781

What it does:

This will log the correct failing tests message to stdout during cypress run
Will show all remaining tests of the current spec file as Pending (or just not run when running via cypress open)
Will show all remaining tests in spec files following the failure as Pending.

Add this to your plugins/index.js file

let shouldSkip = false;

module.exports = (on) => {
  on('task', {
    resetShouldSkipFlag() {
      shouldSkip = false;
      return null;
    },
    shouldSkip(value) {
      if (value != null) shouldSkip = value;
      return shouldSkip;
    }
  });
}

Add this to your support/index.js file

function abortEarly() {
  if (this.currentTest.state === 'failed') {
    return cy.task('shouldSkip', true);
  }
  cy.task('shouldSkip').then(value => {
    if (value) this.skip();
  });
}

beforeEach(abortEarly);
afterEach(abortEarly);

before(() => {
  if (Cypress.browser.isHeaded) {
    // Reset the shouldSkip flag at the start of a run, so that it 
    //  doesn't carry over into subsequent runs.
    // Do this only for headed runs because in headless runs,
    //  the `before` hook is executed for each spec file.
    cy.task('resetShouldSkipFlag');
  }
});

abort_strategy using cookies

You can stop the tests with an afterEach hook when the state of the current test has failed, then set a cookie that can later be read at the beginning of every spec file that is run to also stop the specfile. Thanks @oskargustafsson https://github.com/cypress-io/cypress/issues/518#issuecomment-508731869

What it does:

This will log the correct failing tests message to stdout during cypress run
Will show all remaining tests of the current spec file as Skipped (or just not run when running via cypress open)
Will show all remaining tests in spec files following the failure as passed and skip the execution of the tests.

Add this to your support/index.js file

switch (Cypress.env('abort_strategy')) {
  case 'run':
    before(function onBeforeEach() {
      // Skips any subsequent specs, if the run has been flagged as failed
      cy.getCookie('has_failed_test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
          Cypress.runner.stop();
        }
      });
    });
  /* fallthrough */
  case 'spec':
    afterEach(function onAfterEach() {
      // Skips all subsequent tests in a spec, and flags the whole run as failed
      if (this.currentTest.state === 'failed') {
        cy.setCookie('has_failed_test', 'true');
        Cypress.runner.stop();
      }
    });
    Cypress.Cookies.defaults({
      whitelist: 'has_failed_test',
    });
    break;
  default:
}

Run either strategy to exit early on the single spec or on the entire run.

cypress_abort_strategy=run cypress run

cypress_abort_strategy=run cypress spec

`afterEach` stop runner

You can stop the tests for each spec file ran with an afterEach hook when the state of the current test has failed. Thanks to @DanH91 https://github.com/cypress-io/cypress/issues/518#issuecomment-373369129

What it does:

This will log the correct failing tests message to stdout during cypress run
Will show all remaining tests as Skipped (or just not run when running via cypress open)
This will not abort the entire run, it will only exit the currently running spec, so all other spec files will run.

You can add this to your support/index.js file for it to be universal across all spec files.

afterEach(function () {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

jakedowns commented 4 years ago

@jennifer-shehane Thank you for compiling this list. It was tough to tell which proposed workarounds were still valid since a lot of comments mentioned them not working on newer/latest version

@drumslave-git thank you for pointing out your workaround, i had overlooked it and I think it's the one I need for my usecase

Sorry for the negative comment, frustrated about other things. Thanks for a great tool and a responsive support community. I hope to contribute something constructive to this project as I learn more about it's ins and outs.

loren138 commented 4 years ago

I have a PR which will bail once a test fails #7892.

The mocha bail flag is a boolean so I've built this as a boolean rather than accepting a count https://mochajs.org/api/mocha. By defaulting count to 1, it would be possible to extend this to accept a count and be backwards compatible in the future.

loren138 commented 4 years ago

Similar to @PierreCavalet above, we are now running our tests one file at a time. This works pretty well except cypress takes several seconds to start up before it starts running tests so with all our test files this wastes several minutes of time on successful runs.

./bootServer & SERVER_PID=$!

npmbin=$(npm bin)
for file in ./tests/e2e/*.test.js
do
  ${npmbin}/cypress run --spec="${file}" || ${npmbin}/cypress run --spec="${file}" // try twice due to flake
  CYPRESS_RESULT=$?
  [ $CYPRESS_RESULT -eq 0 ] || break
done

kill -2 $SERVER_PID

exit $CYPRESS_RESULT

I am also using the afterEach stop runner code from above to skip tests in the current spec once a single test fails.

I would love to find a way to "cache" the cypress bootup or otherwise make that faster to make this more viable with a large number of files.

sesam commented 4 years ago

We run specs through a custom spec-file-packer runner that takes all specs, then if one fails we track it in an array, and finally repeats broken tests from last run up to two more times. That might be an option for others reading this ticket too.

This hack allows us to stay "green" while working slowly on making specs less flaky.

(Next level will be to mock away external APIs which will probably get us to "5 nines".)

dillingham commented 4 years ago

@jennifer-shehane can we have a simple (and somewhat common) test config option?

{
    "baseUrl": "http://app.wip",
    "stopOnFirstFailure": true
}

Lvlynfnt commented 4 years ago

afterEach(function () {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

No longer working using version 5.0. Test stops even if it's passed.

lonewarrior556 commented 4 years ago

Wow this has been highly request feature for 3 years... since Cypress Version: 0.19.2 I'm sure it'll be added any day now...

dvsoukup commented 4 years ago

I was wanting to abort early as well, as often when the first test fails in some spec, we don't want to continue along since the rest of the DOM is likely to not be in a state to allow for the other tests to pass. Also, due to some flakyness once in awhile, it can be useful to use retries. However current suggestion didn't work well with setting retries in the cypress.json. Thus, here's something that seems to work well!

As of version 5, you can set test retries. In cypress.json:

    "retries": {
        "runMode": 3,
        "openMode": 0
    },

In my case, I only want to retry in headless mode.

Then, in support/index.js:

function abortEarly() {
    if (this.currentTest.state === 'failed' && this.currentTest.currentRetry() === this.currentTest.retries()) {
        Cypress.runner.stop();
    }
}

beforeEach(abortEarly);
afterEach(abortEarly);

Be sure to use function and not fat arrow syntax, to preserve the state of what is on this. This one will check the currentTest (as defined within the Mocha framework...) current retry count according to what the max is allowed, and only stop the runner at that point. Worked like a champ!

iozz commented 4 years ago

Thanks @dvsoukup it works great. I just use the afterEach() hook.

I've combined your approach with the use of a cookie to be able to avoid the remaining spec files as well. Then, in support/index.js:

export const FAILED_TEST_COOKIE = 'has-failed-test';

// CYPRESS_ABORT_STRATEGY environment variable:
// run: Play the whole run
// spec: Stop on first error
if (Cypress.env('ABORT_STRATEGY') === 'spec') {
  // Do not clear the cookie between tests, to be able to shunt down all remaining spec files
  Cypress.Cookies.defaults({
    preserve: FAILED_TEST_COOKIE
  });
}

before(function () {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
      cy.getCookie(FAILED_TEST_COOKIE).then(cookie => {
      if (cookie && Boolean(cookie.value) === true) {
        Cypress.runner.stop();
      }
    });
  }
});

afterEach(function () {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    // Check if it is the last retry, and if it has failed
    if (this.currentTest.state === 'failed' && this.currentTest.currentRetry() === this.currentTest.retries()) {
      // Set cookie to abort remaining spec files
      cy.setCookie(String(FAILED_TEST_COOKIE), 'true');
      // Stop the current spec file
      Cypress.runner.stop();
    }
  }
});

ryanjyost commented 4 years ago

@iozz 's solution seems to do the trick, but I needed to change currentRetry() to _currentRetry and retries() to _retries.

I also replaced Cypress.runner.stop() in before hook to this.skip() so it's clear that we are skipping tests after the failed one.

My project has cypress": "^5.0.0"

export const FAILED_TEST_COOKIE = 'has-failed-test'

// CYPRESS_ABORT_STRATEGY environment variable:
// run: Play the whole run, spec: Stop on first error
if (Cypress.env('ABORT_STRATEGY') === 'spec') {
  // Do not clear the cookie between tests, to be able to shut down all remaining spec files
  Cypress.Cookies.defaults({
    preserve: FAILED_TEST_COOKIE,
  })
}

before(function() {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    cy.getCookie(FAILED_TEST_COOKIE).then((cookie) => {
      if (cookie && Boolean(cookie.value) === true) {
        this.skip()
      }
    })
  }
})

afterEach(function() {
  if (Cypress.env('ABORT_STRATEGY') === 'spec') {
    // Check if it is the last retry, and if it has failed
    if (
      this.currentTest.state === 'failed' &&
      this.currentTest._currentRetry === this.currentTest._retries
    ) {
      // Set cookie to abort remaining spec files
      cy.setCookie(String(FAILED_TEST_COOKIE), 'true')
      // Stop the current spec file
      Cypress.runner.stop()
    }
  }
})

jpike88 commented 3 years ago

I feel I am taking crazy pills reading this thread. It's like a car without brakes. Instead of stabbing at workarounds that may break as soon as tomorrow, why isn't there a clear, single flag to just fix this? I assume it's pretty damn easy to implement. We will fork if this isn't going to be addressed properly

csvan commented 3 years ago

I assume it's pretty damn easy to implement

That's pretty rude. If you don't understand the internals, don't make assumptions about how easy something is to do.

But if you are sure, why not submit a PR?

leettaylor commented 3 years ago

I assume it's pretty damn easy to implement

That's pretty rude. If you don't understand the internals, don't make assumptions about how easy something is to do.

But if you are sure, why not submit a PR?

Maybe because there's already a PR with the functionality (#7892) that has been waiting deeper evaluation from the Cypress product team since July.

jennifer-shehane commented 3 years ago

The option to abort on first (or a specified) number of failures is still under consideration and we are gathering interest from this thread to help us weigh this feature against others. We are reading the feedback and it does help in our evaluation of what to work on.

Delivering this requires work across both our Test Runner and Dashboard teams in order to deliver a good experience for both of our products. So it's a bit more complex than the initial PR that was proposed.

Priorities can change as we move forward, but our Roadmap shows the larger features that we plan to work on in both products: http://on.cypress.io/roadmap

biiiipy commented 3 years ago

The option to abort on first (or a specified) number of failures is still under consideration and we are gathering interest from this thread to help us weigh this feature against others. We are reading the feedback and it does help in our evaluation of what to work on.

Delivering this requires work across both our Test Runner and Dashboard teams in order to deliver a good experience for both of our products. So it's a bit more complex than the initial PR that was proposed.

Priorities can change as we move forward, but our Roadmap shows the larger features that we plan to work on in both products: on.cypress.io/roadmap

Thank you, we really needed more PR BS here...

mattvb91 commented 3 years ago

Is it just me or are comments getting deleted? There was a comment here explaining this issue would cut into the "time for runs" business model of cypress if it got merged.

Edit: I would just like to add in that if it is a business case decision that from my side its a valid reason to choose against it but it could just be communicated that way.

jennifer-shehane commented 3 years ago

The Cypress team has not deleted any comments in this thread. If comments are deleted in GitHub there is a timeline event that is posted within the thread that shows that a comment was deleted. This would be viewable to everyone.

From GitHub docs: https://docs.github.com/en/free-pro-team@latest/github/building-a-strong-community/managing-disruptive-comments#deleting-a-comment

People are able to edit/delete their own comments however, and you should be able to see an edit history of any edits on each individual comment.

lonewarrior556 commented 3 years ago

People are able to edit/delete their own comments however, and you should be able to see an edit history of any edits on each individual comment.

And yet it is gone

QuentinFarizon commented 3 years ago

I'm surprised such a great testing tool doesn't have this baked-in from day one ...

jennifer-shehane commented 3 years ago

@lonewarrior556 yes, people can delete their own comments. This would remove the comment entirely without showing it in the history. If it is deleted by a our team it shows the history of all edits/deletions.

dillingham commented 3 years ago

I deleted my comment lol Didn't seem accurate after looking at their pricing.

csvan commented 3 years ago

@biiiipy can you teach me how to solve other problems through nothing but whining?

lonewarrior556 commented 3 years ago

@lonewarrior556 yes, people can delete their own comments. This would remove the comment entirely without showing it in the history. If it is deleted by a our team it shows the history of all edits/deletions.

Wasn't trying to accuse anyone. Soo should we all delete out comments regarding this or would this loop continue?

jennifer-shehane commented 3 years ago

Please be mindful of our Code of Conduct when interacting with other commenters in our issues. https://github.com/cypress-io/cypress/blob/develop/CODE_OF_CONDUCT.md We'd like to keep the thread open and on the topic of specific requests for the feature or existing workarounds. Thanks!

csvan commented 3 years ago

@vesper8 it has already been explained why. https://github.com/cypress-io/cypress/issues/518#issuecomment-733573085

Also, that accusation was deleted by @dillingham after actually checking their pricing model.

jakedowns commented 3 years ago

well, if the interest level isn't clear enough, consider this a 👍 from me. maybe a uservoice page with pure votes, or an upvote system on your own roadmap page would be useful :)

that said, i've been using the afterEach stop just fine

javierbrea commented 3 years ago

If anyone is interested, I have just published a Cypress plugin that can do the trick while the Cypress team works on this feature. The solution is based in some of the proposals I found on this thread, specially those from @DanH91 @dvsoukup @dwelle @jennifer-shehane, etc. so thanks to everybody.

I have added e2e tests to the package, and tested it with Cypress versions from v5.0.0 to v5.6.0 and it works well. I am going to test it also with Cypress v6.x versions, and will add the correspondent tests, so I hope it can be stable enough until the Cypress team adds native support for this feature.

Github: https://github.com/javierbrea/cypress-fail-fast NPM: https://www.npmjs.com/package/cypress-fail-fast

I hope it might help.

fractalf commented 3 years ago

This works for me..

describe('Dummy', () => {
    before(() => {
        const condition = true
        if (condition) {
            throw new Error("Stop the press")
        }
    })
    it('Dummy test', () => {
        cy.wait(60000)
    })
})

csvan commented 3 years ago

@fractalf it works because you are throwing in the Before hook, it won't work if you throw in a test (which is what is asked for here).

antonioiksi commented 3 years ago

Similar to @PierreCavalet above, we are now running our tests one file at a time. This works pretty well except cypress takes several seconds to start up before it starts running tests so with all our test files this wastes several minutes of time on successful runs.
./bootServer & SERVER_PID=$!

npmbin=$(npm bin)
for file in ./tests/e2e/*.test.js
do
  ${npmbin}/cypress run --spec="${file}" || ${npmbin}/cypress run --spec="${file}" // try twice due to flake
  CYPRESS_RESULT=$?
  [ $CYPRESS_RESULT -eq 0 ] || break
done

kill -2 $SERVER_PID

exit $CYPRESS_RESULT
I am also using the afterEach stop runner code from above to skip tests in the current spec once a single test fails.

I would love to find a way to "cache" the cypress bootup or otherwise make that faster to make this more viable with a large number of files.

For windows with Powershell, it is like that:

param (
  [string]$TestDir = $(throw "-TestDir is required.")
)
write-output "TestDir is $TestDir"

$ROOT = $pwd.Path
$CYPRESS_DASHBOARD_KEY = "YOUR_KEY"
$CYPRESS = ".\node_modules\.bin\cypress"

$TEST_DIR = $TestDir

$FILES = Get-ChildItem -Path $ROOT\$TEST_DIR\*.spec.js
foreach ($file in $FILES) {
  # echo $file.FullName
  echo "file.Name=$($file.Name)"
  $FILE_NAME = $file.Name
  $RUN_TEST = "$CYPRESS run --browser chrome --spec '${file}' --record --key $CYPRESS_DASHBOARD_KEY"
  # echo $RUN_TEST
  # Invoke-Expression "$RUN_TEST > log_$FILE_NAME.log"
  Invoke-Expression $RUN_TEST
  echo "LASTEXITCODE=$LASTEXITCODE"
  if ($LASTEXITCODE -ne 0) { break }
}

exit $LASTEXITCODE

cypress-io / cypress