REOPEN: Exit with code 1 when tests fail (for CI purposes) #49

[trikonasana]

Hi,

the exit code 1 for failed tests does not work anymore. seemingly with Version 0.3.3

Best regards tri

[Theyssen]

Hi,

I am having the same problem. In the logs from Travis CI you can see the test fails, but Travis thinks it passed because the exit code is 0.

[dmechas]

Any update?

[stephane-ruhlmann]

@DavertMik I think this feature is essential for continuous integration purpose, any idea on how we could make it happen?

[DavertMik]

I think it is fixed. At least this is how tests on Travis are executed

https://travis-ci.org/Codeception/CodeceptJS/jobs/157728265#L939

[DavertMik]

Btw, the exit is happening here https://github.com/Codeception/CodeceptJS/blob/master/lib/listener/exit.js

[erdem]

I got the same issue on my project. When the test fails, it doesn't exit with 1. That's why automated tests never fail on Travis. Is anyone has any idea why that happens?

My command: ./node_modules/.bin/codeceptjs run --steps --verbose;

Config:

{
  "tests": "./src/__tests__/plugins/register-form/acceptance/*_test.js",
  "timeout": 10000,
  "output": "./src/__tests__/plugins/register-form/acceptance/output/",
  "helpers": {
    "WebDriverIO": {
      "url": "http://localhost:3333",
      "browser": "chrome"
    }
  },
  "include": {
    "I": "./src/__tests__/plugins/register-form/acceptance/steps_file.js"
  },
  "bootstrap": "./src/__tests__/plugins/register-form/acceptance/app-server.js",
  "teardown": "./src/__tests__/plugins/register-form/acceptance/app-server.js",
  "mocha": {},
  "name": "acceptance"
}

I am using 1.0.0 version. Thanks

[APshenkin]

@erdem Can you please update to 1.0.2 version? Also verbose logs will be helpfull

[erdem]

@APshenkin I updated 1.0.2 version but it didn't work. You can see full logs here.

Thanks

[APshenkin]

@erdem I don't see that global.result event is not emitted. It should after results printing. https://github.com/Codeception/CodeceptJS/blob/644b957440debcaa2e242ade9559dfec8c3bd859/lib/codecept.js#L112

Is this problem reproducible locally?

Looks like there is an error in your teardown file, that throws err https://travis-ci.org/nhsuk/register-with-a-gp-beta-web/builds/272956412#L2912

and doesn't complete done callback.

Can you please try to remove teardown and test it again?

[erdem]

@APshenkin Sorry for the delayed answer. Yes, I reproduced the same issue on my local. Also, I called that done callback in my teardown function, so "global.result" event must be emitted. On the other side, I removed bootstrap and teardown attributes from my codecept.json config file and Travis still not fail.

The project latest build without teardown and bootstrap: https://travis-ci.org/nhsuk/register-with-a-gp-beta-web/builds/274653315

You can check my bootstrap and teardown functions here.

Thanks

[APshenkin]

@erdem I found two issues in your configurations:

1. in package.json "test:acceptance_local": "export ACCEPTANCE_TEST=1;babel-node ./node_modules/.bin/codeceptjs run --steps --verbose; echo $?;unset ACCEPTANCE_TEST", this command completes several commands: set ENV ACCEPTANCE_TEST, run tests, print exit code and unset ENV ACCEPTANCE_TEST. Because we don't stops process execution after test fail (we just set process exit code to 1), other commands (print and unset) override process exit code.

So you have to move this to other step in your travis config

2. In your teardown script you Just stop codeceptjs with exit code 0, because of this you get process exit code 0

I test your project after these changes and it work as expected

[erdem]

@APshenkin Thank you for investigate. Yep, I missed that yarn command part, it hacks for something. The command just worked fine.

[APshenkin]

Ok, will close this issue. If somebody will face this issue again, please open new issue for it