search

MishaKav / pytest-coverage-comment

Comments a pull request with the pytest code coverage badge and full report
MIT License
182 stars 60 forks source link

Can I have the action fail if tests fail? #128

Closed BradLucky closed 1 year ago

BradLucky commented 1 year ago

I have noticed that teeing the output of pytest in order for pytest-coverage-comment to leave a comment means that test failures do not cause the action to show as failed in Github. In fact, it hasn't "failed," since the exit code is swallowed by teeing the output.

I have found two alternatives to address this situation:

  1. Don't tee the output and simply use an XML report as described in the docs. The downside to this is that a comment is not left on the PR.
  2. Add another step after the commenter testing if there were any errors or failures and return an exit code if so. This adds a step to the workflow, but leaves a comment and still reports the action as failed.

Is there a way built in to pytest-coverage-comment to cause the action to fail if there are failures? If not, I feel it would be a worthwhile option. In fact, it's the expected behavior IMHO. I did not know when implementing this commenter that my PRs would suddenly "go green" even when tests failed.

pamelafox commented 1 year ago

I was just looking for this too! Looks like it was discussed in https://github.com/MishaKav/pytest-coverage-comment/issues/69 and workarounds given there. I'll try those, though I do think the intuitive thing is an option that defaults to True for failing the step on test failure.

MishaKav commented 1 year ago

teeing the output is like a simple "workaround" to get the comment-report without modifying the workflow too much. The best thing to do is to use the pytest-xml-coverage-path (--cov-report "xml:coverage.xml), in this way you will get the report-comment and fail the action when the pytest will fail or the solutions that provided #69