feature request: support coverage.py's dynamic contexts?

[jclerman]

Description

Thank you for creating this project! It's especially nice to have pytest-esque support for unittest test-parameterization, for projects that are using unittest.

I've have found the "dynamic contexts" feature of coverage.py to be extremely useful - it tracks which tests exercised which lines of code in a project. Unfortunately though, something about the way dynamic contexts are determined seems to be incompatible with tests that are parameterized using this library.

I am not quite sure of the best way forward, but it would be really great to see parameterized tests showing up in the dynamic context documentation that coveragage.py can produce (I use it to generate HTML coverage documentation).

Might you be interested in updating unittest-parametrize to (optionally?) generate dynamic context info that's compatible with coverage.py?

The key parts of a .coveragerc that enables this feature are:

[run]
dynamic_context = test_function

[html]
show_contexts = True

[adamchainz]

Yeah that seems like a fair extension. I won't have time to work on it but if you can make a PR that would be welcome.

It would be good to also see if and how pytest handles it.

[jclerman]

I think pytest has a bit of an advantage since parameterization is built into pytest itself. That means there are only two libraries that need to cooperate (pytest & coverage), and the pytest-cov plugin handles that cooperation by (effectively) extending pytest. Per the coverage documentation I linked above:

The pytest plugin pytest-cov has a --cov-context option that uses this [calls the Coverage.switch_context() method to set the context explicitly] to set the dynamic context for each test.

Without having yet looked at the unittest-parametrize code, I am thinking that that approach would be the best thing to do here as well. It would add a coverage dependency, so should probably be implemented as an extra, unless you are OK adding coverage as a base dependency.

[adamchainz]

Yes, that looks like the right way. But does it work with parametrized tests in pytest?

And yeah, make it an optional dependency. I'm not sure using the "extras" mechanism will be helpful, we can just try import coverage, and if that succeeds, use it.

[adamchainz]

Just thinking more, this seems like something that should live in the unittest runner, so it can run for every test method. Then it should “just work”, no? unittest-parametrize modifies the test class before the runner ever reads it, so the generated parametrized test methods look like regularly defined ones.

[jclerman]

I have been thinking about this a lot, and trying to understand what the issue is that causes this problem in the first place. coverage has built-in support for computing a "dynamic context" for each unittest test-case, and I don't really understand, as I think you're saying too, why that's not working in the case of unittest-parameterized-modified test-cases.

The approach that pytest-cov uses is not quite appropriate for unittest-testcase since pytest-cov is a pytest plugin that actually starts coverage, so it has access to the coverage.Coverage() object at runtime, so it can call Coverage.switch_context().

If the actual root problem were clear, one option would be to register a coverage "Dynamic Context Switcher" plugin that provides a method to compute dynamic context - but that method would, as far as I can tell, essentially recapitulate what coverage is already doing by default, in coverage.context.should_start_context_test_function() (so we shouldn't need a plugin).

I wonder if @nedbat might be able to weigh in here and explain what the actual root of the problem might be.