Green reports no tests, but runs them??

[MinchinWeb]

I'm trying to use Green on Travis-CI and Coveralls.io.

I can run the tests localling without issue. I'm running Windows 10 x64, Python 3.5.0, Green 2.0.7, Coverage 4.0.1. I use the command green -vvr from the projects root directory. The (local) output:

test.Test_Setup
  Test_Setup
s   test_true_is_false -- Proof 'skip' is working
F   test_true_is_false_2
.   test_version
.   test_we_live

Skipped test.Test_Setup.Test_Setup.test_true_is_false - Proof 'skip' is working

Failure in test.Test_Setup.Test_Setup.test_true_is_false_2
  File "c:\program files\python 3.5\lib\unittest\case.py", line 58, in testPartExecutor
    yield
  File "c:\program files\python 3.5\lib\unittest\case.py", line 597, in run
    testMethod()
  File "S:\Documents\GitHub\verdant-barnacle\test\Test_Setup.py", line 24, in test_true_is_false_2
    self.assertTrue(False)
  File "c:\program files\python 3.5\lib\unittest\case.py", line 674, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

Name                           Stmts   Miss  Cover   Missing
------------------------------------------------------------
verdant_barnacle\__init__.py       3      1    67%   5

Ran 4 tests in 0.621s

FAILED (failures=1, passes=2, skips=1)

However, when I push the code to Github (with the test directory as a sibling to the main code directory), Travis-CI picks it up and runs it, and fails to any tests, but still generates and sends a coverage report to Coveralls.io (!?). The output from Travis-CI is:

$ green verdant_barnacle -vvr
Name                           Stmts   Miss  Cover   Missing
------------------------------------------------------------
verdant_barnacle/__init__.py       3      1    67%   5
Ran 0 tests in 0.146s
No Tests Found
The command "green verdant_barnacle -vvr" exited with 0.

However (again), if the test directory is a 'sibling' to the main code directory, green still fails to find the tests, but now fails to generate a coverage report too. The output from Travis-CI is:

$ green -vvr
Coverage.py warning: No data was collected.
Name    Stmts   Miss  Cover   Missing
-------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.4.2/bin/green", line 9, in <module>
    load_entry_point('green==2.0.7', 'console_scripts', 'green')()
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/green/cmdline.py", line 75, in main
    result = run(test_suite, stream, args, testing)
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/green/runner.py", line 130, in run
    result.stopTestRun(testing)
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/green/result.py", line 380, in stopTestRun
    self.args.cov.report(file=self.stream, omit=self.args.omit_patterns)
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/coverage/control.py", line 966, in report
    return reporter.report(morfs, outfile=file)
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/coverage/summary.py", line 109, in report
    raise CoverageException("No data to report.")
coverage.misc.CoverageException: No data to report.
The command "green -vvr" exited with 1.

Any ideas how to fix this?

P.S. The code of this is on Github at https://github.com/MinchinWeb/verdant-barnacle

[CleanCut]

That is very strange behavior. I don't have any ideas off the top of my head. I'll see if I can look into it this afternoon or Monday. Let me know if you make any progress with it before then.

[CleanCut]

@MinchinWeb I just released Green 2.2.0, which has a Python 3.5 fix in it. Could you check real quick to see if it happens to also fix this issue?

[MinchinWeb]

@CleanCut Yes and no. Green has stopped 'blowing up' if there's no coverage data (yeah!), but it still doesn't seem to find my tests on Travis-CI.

Here's the link to the Travis build -> https://travis-ci.org/MinchinWeb/verdant-barnacle/jobs/87557302 There should be 5 tests found, and one of them should fail. This works as expected on my end on my local (Windows) machine.

Note this isn't just a Python 3.5 thing, but across all versions of Python: 2.7, 3.3, 3.4, and 3.5.

[e3krisztian]

The test file has a mixed case name and has underscore as well: Test_Setup.py. This naming looks strange to me - might it contribute to the unexpected behavior?

[MinchinWeb]

@krisztianfekete I stripped out all the underscores and made everything lowercase, but it doesn't seem to change anything.

Here's the link to the Travis build -> https://travis-ci.org/MinchinWeb/verdant-barnacle/jobs/87730614

[e3krisztian]

Only the file name was problematic (Test_Setup.py -> test_setup.py), internal (python code) renames were unnecessary.

The travis build log is no surprise this time, as you were running green against module/package tests in a directory where you have only README.md and verdantbarnacle.

Looking at your first build I do think it is a file case related thing (Windows' file systems are notoriously case ignoring, while travis is running on linux): no tests are run and coveralls has only a coverage for your __init__.py - not any tests.

I recommend fixing .travis.yml, so that it runs green -rvv without any other parameters.

[CleanCut]

Good call, @krisztianfekete -- these types of Windows-specific issues are difficult for me. I had the good fortune to stop being a Windows user in 2003.

Does that mean we can close this issue? Filename case mismatches are beyond the scope of green.

[MinchinWeb]

Thanks @krisztianfekete, fixing .travis.yml as well as the lowercase names seems to have fixed it. I feel comfortable saying that was starting the filenames capitalized that was causing this issue.

@CleanCut: is the test discovery code local to green, or pulled in from unittest? If it's a simple fix, it might have sense to make that part of the code case-insensitive. For Windows users, this issue (and it's fix) is entirely non-obvious. In fact, on Windows GUI, if you try and rename a file but the only thing you are changing is the case of a letter, it will ignore the renaming and return the filename as originally capitalized! Furthermore, with git on Windows, if the only change is in the capitalization of a filename, it isn't recognized as a commit-able change.

[CleanCut]

Most of discovery is done by running the discovery in your own python's unittest.

[e3krisztian]

@MinchinWeb you're welcome!

Windows is known to be problematic in Python (e.g. most of packaging are about windows and binaries).

Python is a case sensitive language, your directory/file names become package/module names, this is how it works PEP-0235, look at Proposed Semantics, case B:

search sys.path for the first case-sensitive match; raise
       ImportError if none found

Moreover, the consensus coding style says

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.