Collecting coverage when testing a pytest plugin with `pytest.main()`

[fjarri]

Summary

I have a package that includes a pytest plugin (and, to make things more complicated, the test suite for the package uses that plugin). I want to run some tests for the plugin calling the test suite programmatically with pytest.main() and I want the resulting coverage to be added to that of the whole test suite run.

Expected vs actual result

If I invoke pytest.main() with --cov-append (which seemed logical to me at first), the coverage of the plugin tests is not added to the final report. I am guessing because --cov-append is supposed to only append coverage from sequential runs?

If I invoke pytest.main() with --no-cov, the coverage of the plugin tests is included in the final report, but for every testcase executed inside pytest.main() (and there are quite a few of those), a temporary file with a name like .coverage.<HOSTNAME>.<NUMBER>.<NUMBER> is created, which is quite annoying (I see it in my IDE), slows down the test suite, and also triggers the Dropbox folder where the project is located. Plus it seems strange that any kind of coverage files should be created at all with --no-cov in place.

Reproducer

The second case can be reproduced by the following setup (in a single folder):

pytest.ini:

[pytest]
addopts = -v --cov=mymodule --cov-report=html --cov-report=term -m "not inner_test"
markers =
    inner_test: marks tests used for testing custom fixtures (called in a sandbox only)

mymodule.py:

def func():
    return True

test_mymodule.py:

import mymodule
import pytest
import time

@pytest.mark.inner_test
@pytest.mark.parametrize('param', range(100))
def test_something(param):
    assert mymodule.func()
    time.sleep(0.1)

def test_outer():
    pytest.main(['-m', 'inner_test', '--no-cov'])

When you run py.test you can see a hundred of temporary files appearing.

Versions

Python 3.7.5 Pytest 5.3.2 Pytest-cov 2.8.1

Code

The actual project is https://github.com/fjarri/grunnur Namely, the plugin in question is https://github.com/fjarri/grunnur/blob/master/grunnur/pytest_plugin.py and the tests for it are in https://github.com/fjarri/grunnur/blob/master/test/test_mocked/test_pytest_plugin.py

[ionelmc]

So the temp .coverage.123123.123.123 files are there by design, and you'd get many of them if you use subprocesses/multiprocessing. There isn't anything I can do about that. The best I've got so far is to implement better cleanup in v2.9.0.

[fjarri]

Why are they created at all if the coverage is off? Is it possible to at least specify a custom directory for them (or put them in a temporary directory by default)?

There doesn't seem to be any problems with cleanup though, all these files are gone when the test run ends.

[ionelmc]

Well you haven't hit the cleanup bug yet :-)

They can't be in /tmp cause how would you know to wich project they belong? I guess there could be some filename encoding scheme to filter them but yeah, you're the first to ask this.

Why aren't you using git/hg? Seems to me like you're really trying to get all your money's worth out of dropbox with your setup : -)

[fjarri]

They can't be in /tmp cause how would you know to wich project they belong?

I thought they are only needed for a single run, so a new folder could be created each time? Anyway, why are they created if no coverage is collected? Could you point me to the relevant place in the codebase?

Why aren't you using git/hg?

I am, as evidenced by the links to my project on github :) I develop simultaneously on two machines, and often need to try a new change on a different machine before committing it (it's a videocard thing, different platforms can have their own quirks)

[ionelmc]

Ok so if I'd run your project with --no-cov I'd still get some .coverage.123123 files created?

[fjarri]

Only the tests run via pytest.main(['--no-cov']) (test_something() in the small example above) create these files. --no-cov doesn't seem to do that when just used in the command line.

[ionelmc]

Does it still happen if you use --no-cov both on the outside and on the inner pytest.main call? I suspect there's some global state somewhere in pytest. I wouldn't run pytest in pytest inprocess anyway. Using suprocesses is more reliable.

[fjarri]

Does it still happen if you use --no-cov both on the outside and on the inner pytest.main call?

Nope.

I wouldn't run pytest in pytest inprocess anyway. Using suprocesses is more reliable.

Perhaps it is. The problem in my case is that to test the Pytest plugin properly I need to mock some things first. So I would have to write a special script that does that and then calls pytest.main(), and call this script in a subprocess, and then make sure that the coverage data is joined with that of the main run. It may work, but so far it is easier for me to live with some temporary files.