Coverage: warning "module-not-measured"

florealcab commented 4 years ago

Hello,

Thank you for your good work, tests take 2x less times to run with it comparing to unittest.

But I have a problem. I run with this command: unittest-parallel -t . -s . -p 'test_*.py' --coverage --coverage-source XXX

where XXX is the name of my package

Just before the coverage report, i have this warning: Coverage.py warning: Module XXX was previously imported, but not measured (module-not-measured)

Is there a way to fix this warning?

craigahobbs commented 4 years ago

I'm glad to hear that you find unittest-parallel useful! Sadly, I was unable to reproduce the issue. I'm running Ubuntu 20.04. Here's my venv setup:

python3 -m venv env
env/bin/pip install -U pip setuptools wheel
env/bin/pip install coverage unittest-parallel

Here are my versions:

$ env/bin/pip freeze
coverage==5.2.1
pkg-resources==0.0.0
unittest-parallel==0.8.7

I setup the following directory tree:

.
├── mypkg
│   ├── __init__.py
│   └── mypkg.py
└── tests
    ├── __init__.py
    └── test_mypkg.py

The "__init__.py" files are empty. "mypkg.py" contains the following:

def myfunc():
    return 42

"test_mypkg.py" contains:

import unittest

from mypkg.mypkg import myfunc as myfunc

class TestMypkg(unittest.TestCase):
    def test_myfunc(self):
        self.assertEqual(myfunc(), 42)

I then ran unittest-parallel like so:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source mypkg

I get the following output with no warnings:

test_myfunc (tests.test_mypkg.TestMypkg) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Ran 1 tests

Name                Stmts   Miss  Cover
---------------------------------------
mypkg/__init__.py       0      0   100%
mypkg/mypkg.py          2      0   100%
---------------------------------------
TOTAL                   2      0   100%

Total coverage is 100.00%

florealcab commented 4 years ago

In my case, module is installed (pip install . in the source folder) inside the virtualenv. Maybe it's why it warns? With your test I have no warning too.

But if I use official unittests with coverage, I have no warning with my module

craigahobbs commented 4 years ago

Can you try disabling the warning using a .coveragerc file?

https://coverage.readthedocs.io/en/coverage-5.2.1/cmd.html?highlight=warnings#warnings

ferdnyc commented 2 months ago

@florealcab @craigahobbs

(Four years later...) I was able to reproduce this, but only by making some modifications to the example setup @craigahobbs provided above.

The changes I made were:

Move mypkg into a source directory:
```
mkdir src
mv mypkg src/
```

Add a pyproject.toml to make it installable:

[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = 'mypkg'
version = '0.1'
description = "My Package"
license = {text = 'MIT No Attribution License (MIT-0)'}
requires-python = ">= 3.8"

[project.optional-dependencies]
tests = [
    'unittest-parallel',
]

Update the module code to make it import itself when loaded:
```
# src/mypkg/__init__.py

from .mypkg import *
```

Adjust the test code to match:

# tests/test_mypkg.py
import unittest
from mypkg import myfunc

class TestMypkg(unittest.TestCase):
    def test_myfunc(self):
        self.assertEqual(myfunc(), 42)

Install mypkg into the current venv (doesn't really matter if it's an editable install or one built by setuptools, same behavior either way):
```
pip install -e '.[tests]'
```

Run unittest-parallel

$ unittest-parallel -vv --coverage-branch \
 --coverage-source mypkg
Running 1 test suites (1 total tests) across 1 processes

test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ...
test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ... ok
...venv/lib64/python3.12/site-packages/coverage/inorout.py:519: 
CoverageWarning: Module mypkg was previously imported, but not
measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")

----------------------------------------------------------------------
Ran 1 test in 0.205s

OK

Name                    Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------
src/mypkg/__init__.py       1      0      0      0   100%
src/mypkg/mypkg.py          2      0      0      0   100%
---------------------------------------------------------
TOTAL                       3      0      0      0   100%
Total coverage is 100.00%

It seems to be the relative star import in the package __init__.py that causes the problem, coupled with the package being installed. (I couldn't reproduce with a local, non-installed package tree that contained a star import. Only once it was installed.)

And since that's pretty terrible code anyway, it's probably not unfair to just say, "Yeah, don't do that, it's ugly and it confuses coverage.py."

ferdnyc commented 2 months ago

It doesn't actually have to be a star import. I just confirmed that the same behavior occurs if src/mypkg/__init__.py contains this:

from .mypkg import myfunc

The key factor seems to be that the package imports the part of itself that's being tested from its __init__.py, as soon as the top-level package definition is imported into sys.modules, which I guess happens before unittest-parallel's _coverage wrapper is able to trigger the module's import under the controlled conditions it expects.

The src/ paths in the unittest output are just because I happened to do an editable install for that test. If it's performed with a standard pip install '.[tests]' instead, same behavior (including the Coverage warning) but now referencing the installed path:


$ pip install '.[tests]'
$ unittest-parallel -vv --coverage-branch --coverage-source mypkg
Running 1 test suites (1 total tests) across 1 processes

test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ...
test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ... ok
...venv/lib64/python3.12/site-packages/coverage/inorout.py:519:
CoverageWarning: Module mypkg was previously imported, but not 
measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")

----------------------------------------------------------------------
Ran 1 test in 0.180s

OK

Name                                                      Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------------------
venv3.12/lib/python3.12/site-packages/mypkg/__init__.py       1      0      0      0   100%
venv3.12/lib/python3.12/site-packages/mypkg/mypkg.py          2      0      0      0   100%
-------------------------------------------------------------------------------------------
TOTAL                                                         3      0      0      0   100%
Total coverage is 100.00%

ferdnyc commented 2 months ago

(Last update, I swear...)

Relative imports are also not required. Same warning if src/mypkg/__init__.py contains:

from mypkg.mypkg import myfunc

craigahobbs commented 2 months ago

@ferdnyc will take a look next week

craigahobbs commented 2 months ago

OK, I've reproduced it with your changes above and the following command:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source mypkg

The comment above the warning line in the coverage source for inorout.py is informative:

        # The module was in sys.modules, and seems like a module with code, but
        # we never measured it. I guess that means it was imported before
        # coverage even started.

The --coverage-source option takes a source directory or a module name. Since we've moved our package source to "src/mypkg", the --coverage-source option above ("mypkg") refers to the mypkg that is installed in the virtual environment, which is what triggers the warning.

If I change the unittest-parallel command to refer to the package source directory ("src/mypkg"), the warning goes away:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source src/mypkg

Does that fix it for you?

florealcab commented 2 months ago

If I do that coverage is not well measured and I have this new warning instead:

CoverageWarning: No data was collected. (no-data-collected) self._warn("No data was collected.", slug="no-data-collected")

craigahobbs commented 2 months ago

I don't see that warning. I added a test repo for our test case - please update as necessary:

https://github.com/craigahobbs/unittest-parallel-issue-3

It has a makefile to build the venv and run the tests:

Reproduce the original warning

make test

No warning

make test2

Cleanup venv, etc

make clean

florealcab commented 1 month ago

Maybe the problem was I am using setup.py instead of pyproject.toml to configure my package. I can try to switch.

With your test, I see no warning with test2 as expected. I missed the invitation, it has expired now. I will try to switch to toml and if it does not change anything I will try to make a minimal test case that reproduce my problem

ferdnyc commented 1 month ago

The src/mypkg version fails if you've done a non-editable install, in which case coverage does indeed produce a "no-data-collected" error, because src/mypkg is no longer being imported by the test code:

$ build/env/bin/pip uninstall 'mypkg'
 [...]
$ build/env/bin/pip install '.[tests]'
 [...]
$ build/env/bin/unittest-parallel -t . -s tests --coverage --coverage-source src/mypkg
.../build/env/lib64/python3.12/site-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
Running 1 test suites (1 total tests) across 1 processes
.../build/env/lib64/python3.12/site-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

----------------------------------------------------------------------
Ran 1 test in 0.220s

OK

Name                    Stmts   Miss  Cover
-------------------------------------------
src/mypkg/__init__.py       1      1     0%
src/mypkg/mypkg.py          2      2     0%
-------------------------------------------
TOTAL                       3      3     0%
Total coverage is 0.00%

ferdnyc commented 1 month ago

If you run the above with PYTHONPATH=$(pwd)/src prepended, then it succeeds.

Getting coverage.py to work correctly in a multiprocessing environment is apparently a very difficult thing. The coverage.py docs have a lot to say about that.

ferdnyc commented 1 month ago

The src/mypkg version fails if you've done a non-editable install, in which case coverage does indeed produce a "no-data-collected" error, because src/mypkg is no longer being imported by the test code:

If you run the above with PYTHONPATH=$(pwd)/src prepended, then it succeeds.

...However, that's only for one process. Add additional processes (actually take advantage of multiprocessing), and you'll get warnings even using src/mypkg.

Edit the source/test files as:

# src/mypkg/mypkg.py
def myfunc():
    return 42

def myfunc2():
    return 84

# src/mypkg/__init__.py
from .mypkg import myfunc, myfunc2

# tests/test_mypkg.py
import unittest

from mypkg import myfunc, myfunc2

class TestMypkg(unittest.TestCase):
    def test_myfunc(self):
        self.assertEqual(myfunc(), 42)

class TestMypkg2(unittest.TestCase):
    def test_myfunc2(self):
        self.assertEqual(myfunc2(), 84)

And run with --level=class or --level=test:

$ PYTHONPATH=$(pwd)/src build/env/bin/unittest-parallel \
 -vv --level=test -t . -s tests --coverage \
 --coverage-source src/mypkg
Running 2 test suites (2 total tests) across 2 processes

test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ...
test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ... ok
test_myfunc2 (tests.test_mypkg.TestMypkg2.test_myfunc2) ...
test_myfunc2 (tests.test_mypkg.TestMypkg2.test_myfunc2) ... ok
.../build/env/lib64/python3.12/site-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

----------------------------------------------------------------------
Ran 2 tests in 0.310s

OK

Name                    Stmts   Miss  Cover
-------------------------------------------
src/mypkg/__init__.py       1      0   100%
src/mypkg/mypkg.py          4      0   100%
-------------------------------------------
TOTAL                       5      0   100%
Total coverage is 100.00%

(Edit: And that's also true with an editable install, even using --coverage-source src/mypkg.)

craigahobbs / unittest-parallel

Coverage: warning "module-not-measured" #3