Python3: Green does not de-duplicate test modules missing __init__.py

[sodul]

We have a deep tree that is shared by several teams and once in a while we end up with modules with the same name. This is fine for runtime since the import resolution and namespacing is taking care of that.

How to reproduce:

mkdir -p foo/a/test foo/b/test foo/c/test
touch foo/a/__init__.py foo/b/__init__.py foo/c/__init__.py

Create files test_x.py in the 3 test directories:

import unittest

class TestBax(unittest.TestCase):

    def test_x(self):
        print('test x')

Then run: green -vv foo

A single test will run because all test modules will resolve to test_x.

If we add a init.py to every single test directory green will now de-duplicate the modules and all 3 tests are executed.

Since Python 3 does not require __init__.py files anymore this makes green behave inconsistently with the python documentation. Considering there is a workaround this is non blocking but this should probably be corrected to ensure better consistency with python 3 import behavior.

[CleanCut]

Dang, that's tricky. To restate the problem: Python 3 views this hierarchy as a single implicit namespace package with root bar, while Green sees it as two separate packages both named test_x which happen to reside in different directories (since there's no __init__.py in baz or quux).

bar
├── __init__.py
├── baz
│   └── test_x.py
└── qux
    └── test_x.py

Green relies so much on unittest's internals, that I would have hoped that we would have gained this sort of functionality for free. Alas, I was forced to hack the loading quite a bit to get things to work. Apparently, there are sufficient changes in green/loader.py that we bypass Python's new implicit module handling.

I'm unlikely to get to work on this problem any time soon, so in the meantime I recommend using "regular" packages with __init__.py in them (which are, of course, still supported by both Python and Green) as a simple workaround. That should be doable by anyone who isn't actually stitching together disparate directories into actual namespace packages.

[sodul]

Re-adding the Python 2 style empty __init__.py files is what we have been doing. We even have a check as part of our CI pipeline to ensure that these files are always present.

Note that this is not a problem just for Green but for other tools such as pylint. Not a big deal but it is good to know since other might be surprised that their python 3 code is not fully tested in these cases.

Thanks for all the work on Green.

[CleanCut]

You're welcome! It's fun to hear about folks actually using Green. 😄

[CleanCut]

I'm going to go ahead and close this, as I don't plan to take any action on it. If someone has a clever idea about how to accomplish this, I would love to review a PR or discuss it!