search

CleanCut / green

Green is a clean, colorful, fast python test runner.
MIT License
791 stars 75 forks source link

Return a fail if no tests are found. #123

Closed dougthor42 closed 8 years ago

dougthor42 commented 8 years ago

Summary:

Currently (v2.4.1) green creates an empty test suite if no tests are found and then runs that empty suite, returning passing code. This can cause false-positives for CI and should be changed.

Example:

$ python -m green -vvv -s 1 -W -t --run-coverage emap-convert --debug
2016-05-24 15:32:20     DEBUG Attempting to load target 'emap-convert' with file_pattern 'test*.py'
2016-05-24 15:32:20     DEBUG IGNORED exception: No module named 'emap-convert'
2016-05-24 15:32:20     DEBUG Found 0 tests for target 'emap-convert'
2016-05-24 15:32:20     DEBUG No test loading attempts succeeded.  Created an empty test suite.
Coverage.py warning: No data was collected.
Green 2.4.1, Coverage 4.1, Python 3.4.4

Name    Stmts   Miss  Cover
---------------------------
Ran 0 tests in 0.280s

No Tests Found

Build succeeded

As you can see, this technically failed because the package/module name was incorrect (it should have been emap_convert - note the underscore). However, Green reported a pass.

Discussion:

I would recommend changing the default action to fail if no tests are found, rather than pass. This might cause issues with some (albeit very odd) usage cases, so perhaps it's prudent to add a CLI option --empty-suite-passes or something like that.

CleanCut commented 8 years ago

Hmm, failing for no tests found sounds like reasonable behavior.

CleanCut commented 8 years ago

Included in 2.5.0, just released.