search

nose-devs / nose

nose is nicer testing for python
http://readthedocs.org/docs/nose/en/latest/
1.36k stars 395 forks source link

No indication of interrupted tests #1040

Open mikem23 opened 7 years ago

mikem23 commented 7 years ago

If KeyboardInterrupt is raised during testing, then testing stops immediately, the number of completed tests is reported, and an exit code of zero is returned.

I found PR#682 from a few years back. I certainly see the advantage of seeing the partial results if the tests are interrupted, but I think it should be clear that they are partial results and I do not think the exit code should be zero.

Running on Fedora 24 python2-nose-1.3.7-11.fc24.noarch

mikem23 commented 7 years ago

To illustrate how much of a problem this can be, let me tell you how I discovered this.

We had a unit test that was testing a function's behavior of it was interrupted. The unit test used a mock side effect to raise KeyboardInterrupt during the run.

We refactored the code a bit so that KeyboardInterrupt was trapped later, outside of this function. The unit tests passed, but they should not have. This particular test failed with an uncaught exception (KeyboardInterrupt) and instead of recording a failure, the harness showed sane-looking results and a clean exit status. It shouldn't be this easy for a bad test to effectively disable half the test suite without any indication.