Seems like Green runs the same tests multiple times

[dougthor42]

I have a project that I've been fiddling with. I was trying to decide between the following two project structures:

Structure A (the original structure):

+ ProjectFoler
  + dist
  + doc
  + main_package
    - __init__.py
    - module1.py
    - module2.py
    + tests
      - __init__.py
      - test_module1.py
      - test_module2.py
  - .travis.yml
  - other stuff like dev_requirements.txt, appveyor.yml, setup.py, etc.

Structure B (what I tried moving to):

+ ProjectFoler
  + dist
  + doc
  + main_package
    - __init__.py
    - module1.py
    - module2.py
  + tests             <-- note that this is *not* under the package directory
    - __init__.py
    - test_module1.py
    - test_module2.py
  - .travis.yml
  - other stuff like dev_requirements.txt, appveyor.yml, setup.py, etc.

Before making the change from A to B, nosetests and green worked just fine. My project currently has 23 tests.

After switching to structure B, nosetests worked just fine, while green ran 46 tests - exactly 2x the number of tests there are. Running green in verbose showed that it was still running tests from ProjectFolder\main_package\tests in addition to the tests found in ProjectFoler\tests

So I decided to switch back to structure A. Again, nosetests run 23 tests correctly, but this time green ran 69 tests - 3x! Running in verbose shows that green runs:

1. tests from ProjectFolder\main_package\tests
2. tests from ProjectFolder\tests
3. tests from ProjectFolder\main_package\tests again

Does green save anything? Is there a file I should delete? I've already tried deleting __pycache__ in all folders.

I'm trying to recreate the issue with a simpler project, but have so far been unsuccessful. I'll continue to try and get exact steps to reproduce the issue.

Edit 1

Oh, I forgot relevant information:

• Green version 1.9.4 (1.10 and above gives me different errors and doesn't run)
• Python 3.4.3, 64bit
• Windows 7 Professional
• Running python via WinPython 1.1

[CleanCut]

Green will not save anything at all. It will, however, look at .pyc and .pyo files, as well as the __pycache__ folder as you guessed.

My only guess without more information is that you either have some compiled python bytecode laying around somewhere, or your "WinPython" is affecting something.

What exactly do you mean by "WinPython 1.1"? I googled, and "WinPython" seems to refer to lots of different things. I couldn't find anything that seemed to be associated with a "1.1" version.

Also, what error do you get with 1.10 and above on Windows? I don't use Windows myself, so I rely on a VM for development and a couple AppVeyor builders to make sure things should work for folks. It appears to be passing, so I'm curious what you're seeing. https://ci.appveyor.com/project/CleanCut/green/build/143

[dougthor42]

Figuring out what error I get on 1.10 and 1.11 will have to wait till I get back to work on Monday, but I vaguely recall something about a runner... Anyway, more details on that later.

I've tried clearing my .pyc and .py0 files too to no avail. I've also been unsuccessful at recreating the issue, though it still exists in my current project. Luckily it's not an issue during my travis or appveyor builds, so it's merely an inconvenience rather than a critical issue.

As for WinPython: it's a portable distribution of Python - meaning no installation required. It comes with things like Spyder, PyQt4, and a bunch of scientific packages already installed. Basically, it's a way to run Python without requiring our IT dept. to install it (because they lock us down like criminals). It's similar to a Python X,Y installation in that it's pre-bundled with a buch of sci and data analysis packages.

For all intents and purposes, WinPython works like a standard Python install. The only difference is that it redirects things like user $HOME and does not integrate with the Windows Shell (right-click menu) unless specifically told to.

I was mistaken when I said WinPython 1.1. Actual version is 3.4.3.3; 1.1 is just the WinPython Control Panel version.

[CleanCut]

green will also detect locally packaged versions of a module, like if you have a setup.py and use it to create a .egg directory.

Is your project open source? If I could see what's going on, it would be much easier to find the issue.

[dougthor42]

green will also detect locally packaged versions of a module

That seems to be the case. I use setuptools to create wheels via python setup.py bdist_wheel and that creates a ProjectFolder\build\lib folder in which contains copies of the source files.

Deleting the build folder fixed the issue. Thanks!

Is there a way to exclude folders from the test runner / search?

---

And yes, the project is open source: https://github.com/dougthor42/PyBank. Still in pre-alpha though, so it might not be all that interesting ;-)

As for the error I was getting in versions 1.10 and 1.11: I can't seem to recreate it anymore. I should have tried to recreate that first, before fixing the duplicate-test issue. Ah well. If I see it again I'll be sure to let you know.

[CleanCut]

Ok, I'll go ahead and close this then.

No, there's currently no way to exclude things as an option. You can, however, specify multiple targets to "include" instead of defaulting to the current directory. For example, in your project you could run green pybank instead of green and that would probably avoid catching any build-related artifacts you happened to have created.

[dougthor42]

Ah yes, green pybank. I really should read the docs :-P

Thanks.

[CleanCut]

No problem! I'm glad you're using green. Out of curiosity, why did you personally pick green for your project?

[dougthor42]

Well it really came down to how easy it was to read a verbose output. Having things color-coded makes a big difference in my opinion. I also really like that there's a hierarchy - you can see the file, class, and method/docstring being run.

Right now I don't have many tests, but I've tried helping debug a large project (100s of tests) and it's a pain to look at nosetests -v. I mean, the least they could to is to organize it as a table:

OK   ... checking test 1
OK   ... checking test 2
SKIP ... test 3
FAIL ... test4
OK   ... test5

but instead they put the result after the test's docstring, so things aren't lined up all pretty-like.

So when I was deciding on a runner for this project, it was between nose because it's the de-facto standard and green. Since I have no real need for the fancier things that nose offers, I chose green.

OK that's slightly wrong: I use nose + coverage to auto-upload coverage info to coveralls.io via the command: nosetests --verbose --with-cov --cov pybank --logging-level=INFO in my .travis.yml file.

I know that green can run coverage, but can you also upload to coveralls.io? If so, then I could switch over to green completely instead of running both nosetests and green.

[CleanCut]

The only thing green won't currently do for you is call the coveralls command for you.

You can just do this, though:

green -r pybank
coveralls

I also recommend adding -s 0 to run the tests in multiple processes if they are concurrent-safe.

[dougthor42]

Oh that's right, I'd forgotten that it's the after_success: coveralls command that uploads data, not the --cov flag in nosetests.

So yes, I can stop using nose altogether, that's nice.

---

One issue: Green's coverage doesn't count 0% files against the total while nose does. Is there an option for this?

Nose:

----------- coverage: platform linux, python 3.4.2-final-0 -----------
Name                             Stmts   Miss  Cover
----------------------------------------------------
pybank/__init__                     13      0   100%
pybank/constants                     4      4     0%
pybank/gui                         563    563     0%
pybank/levenshtein_comparisons      59     59     0%
pybank/ofx                         154     95    38%
pybank/parseofx                    477    274    43%
pybank/pbsql                       454    247    46%
pybank/pybank                       23     23     0%
pybank/tests/__init__                2      0   100%
pybank/tests/test_gui                1      0   100%
pybank/tests/test_ofx               32      3    91%
pybank/tests/test_parseofx         101      7    93%
pybank/tests/test_pbsql             58     15    74%
pybank/tests/test_pybank             1      0   100%
pybank/tests/test_utils              1      0   100%
pybank/utils                         1      1     0%
----------------------------------------------------
TOTAL                             1944   1291    34%
----------------------------------------------------------------------

Green:

Name              Stmts   Miss  Cover   Missing
-----------------------------------------------
pybank/__init__      13      0   100%   
pybank/ofx          154     95    38%   36, 228, 264, 271, 278, 319-323, 333, 340-372, 379, 385, 421-426, 454-460, 483, 508, 514-525, 531-532, 535-546, 552-562, 580-581, 590-598, 606-611, 619-631, 634, 642, 646, 650, 674-697, 702
pybank/parseofx     477    274    43%   76, 133-177, 181-188, 195-207, 222-259, 265-268, 275-281, 287-308, 315, 321, 327-357, 363-365, 430, 517-539, 556-559, 580-603, 683-694, 700-712, 733-736, 749-755, 772-775, 783-784, 792-794, 802-806, 814-816, 824-825, 833-837, 845, 866-867, 909-910, 946-960, 969-970, 977-979, 987, 999-1005, 1013, 1021, 1029, 1069-1077, 1085, 1092, 1102-1103, 1274-1285, 1352
pybank/pbsql        454    247    46%   30, 199, 210, 351-358, 438-439, 571-579, 589-590, 623-637, 652-653, 678-679, 693-694, 700, 704-705, 715-716, 721, 726-727, 735, 740, 745, 755, 763, 786-791, 801, 811, 814, 824, 862, 872, 882-884, 890-892, 895-916, 919-920, 923-924, 944-954, 957-975, 978-979, 982-983, 986-987, 991-997, 1000-1009, 1088, 1152, 1159, 1166, 1173, 1177-1178, 1184, 1193-1205, 1214, 1221, 1240-1241, 1270-1274, 1297-1301, 1306, 1311, 1414, 1419-1421, 1425-1445, 1449-1453, 1457-1480, 1488-1511, 1515, 1519-1538, 1552-1555, 1565-1568, 1571-1572, 1576-1579, 1583-1588, 1592-1599, 1622-1628
-----------------------------------------------
TOTAL              1098    616    44%   

[CleanCut]

Hmmm. No. That must be a design difference.

Green will only run coverage on files that you import through your tests. It presumes that if you didn't import it, then you didn't want to test it, so you don't want it covered. (As a nice extra, green also presumes that you don't care about coverage of your actual test files, since that's sort of nonsensical.)

You can get really close if you just import all of your library files in at least one of your test files. (You can write a fairly simple loop that just imports every module in the tree).

I don't have any plans to change that behavior at the moment, though I could be persuaded to change that if you made a nice pull request or found a built-in feature of coverage to trigger that.

[dougthor42]

I'll have to decide which I like better.

On the one hand, it's true I probably don't care about coverage for files that I don't test. I also don't care about coverage of actual test files.

On the other hand, it's a good way to remind me to add tests for those 0% files.

I'm leaning towards your method, but we'll see what the future holds.

Thanks for all the quick replies!

[CleanCut]

No problem!