Unittest CLI does not support test packages very well

[bitdancer]

BPO	15007
Nosy	@ezio-melotti, @merwok, @bitdancer, @voidspace

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature'] title = 'Unittest CLI does not support test packages very well' updated_at = user = 'https://github.com/bitdancer' ``` bugs.python.org fields: ```python activity = actor = 'ezio.melotti' assignee = 'none' closed = False closed_date = None closer = None components = [] creation = creator = 'r.david.murray' dependencies = [] files = [] hgrepos = [] issue_num = 15007 keywords = [] message_count = 3.0 messages = ['162374', '162379', '162385'] nosy_count = 4.0 nosy_names = ['ezio.melotti', 'eric.araujo', 'r.david.murray', 'michael.foord'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue15007' versions = ['Python 3.3'] ```

[bitdancer]

Suppose you have a test package:

testpkg \_init__.py test_mytest.py

If __init__.py is empty and you run

python -m unittest test_pk

no tests are found.

You can get this to work by adding the following boiler plate to __init__.py:

  def load_tests(loader, standard_tests, pattern):
    this_dir = os.path.dirname(__file__)
    if pattern is None:
        pattern = "test*"
    package_tests = loader.discover(start_dir=this_dir,
                                    pattern=pattern,
                                    top_level_dir=this_dir)
    standard_tests.addTests(package_tests)
    return standard_tests

Note that top_level_dir is required to handle specifying more than one test package at a time on the unittest command line. Otherwise the second package gets a loader that already has _top_level_dir set, and so it fails to default to start_dir. I suspect this is also a bug.

This works; it uses discovery to find the tests and returns them using the load test protocol. Other methods could be used to construct the test to add as well. But all have the serious disadvantage that the package name does not appear in the output. Running the above test_pkg command line give results like this with -v:

test_something (test_mytest.Test) ... ok

test_pkg is not mentioned. This is merely annoying when running a single test package, but if you do something like:

python -m unittest -v test_pkg test_pkg2

You can't tell in the verbose output or the test failure output which test package the tests are from.

In summary, unittest needs better support for test packages.

[voidspace]

Whilst I agree in principle... The trouble is that when you do this:

python -m unittest test_pk

What you are saying is "run all the tests from the test_pk module". You *aren't* launching discovery.

This should work:

python -m unittest discover -t . test_pkg

This is more verbose than is ideal. Suggestions for improvements welcome. Having unittest revert to discovery when it is passed a package name that turns out to be empty seems a bit magical (and complex in terms of implementation).

Yes, calling loader.discover inside a load_tests function will mutate that loader - so having discover restore _top_level_dir on exit would be better. Can you post that as a separate issue?

I think there is a separate issue for improving the test failure name (including module) reporting. I'll try and dig out the issue number.

[bitdancer]

Right, I'm not wanting to run discovery from the command line, I'm wanting to run the tests in the package by package name. In my mind, this is exactly parallel to specifying a module name and having unittest automatically discover the TestCase classes in it. We don't have unittest run 0 tests because discovery wasn't invoked when the module name was specified. Why should it be different for a test package? If boilerplate is required in __init__.py to make that happen that's OK, though to my mind not ideal.

Is there some different magic I can put into __init.py that will result in the tests in the package being run such that the package name shows up in the report? Without that, specifying a package name on the unittest command line seems pretty useless. (I mean, to get it to do anything useful, you'd have to be putting all the TestCases in the __init.py, and if you are doing that, why have a package?)

The issue about improving the name output was about making it copy and pasteable (something I would also very much like). The naming issue here is different, about how to get the package name to show up in the fully qualified test name.

I will open another bug for the _top_level_dir issue.