Collection seems to be skipped after a run

[christianbundy]

Steps:

1. Have project with tests in both ./foo and ./bar
2. Run pytest --testmon ./foo
3. Run pytest --testmon ./

Expected: tests from ./bar will be collected and run.

Actual: collected 0 items / 570 deselected / -570 selected

[christianbundy]

I was hoping that pytest --testmon-noselect --collect-only would fix this, but it reports:

5557 tests collected in 6.58s

But running pytest --testmon . after that still only knows about the 570 tests in ./foo.

[pokopt]

Hi! I' trying to reproduce your problem but with no luck. If you run pytest without testmon from 'root' directory does it execute all (foo + bar) tests?

[christianbundy]

Yep! I'll try to see if I can put together a reproducible reproduction.

[christianbundy]

Confirming that a trivial example doesn't reproduce the issue. 👀

[christianbundy]

Here's a reproduction.

docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon core/tests/
===test session starts ===
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: new DB, environment: default
We'd like to hear from testmon users! 🙏🙏 go to https://testmon.org/survey to leave feedback ✅❌
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 577 items                                                                                                                                                                                                                                                                                                  

[redacted test names]

=== 577 passed in 120.43s (0:02:00) ===

That works as expected. But let's try running all of my tests now.

docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon 
================================================================== test session starts ===================================================================
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: changed files: 0, skipping collection of 1830 files, environment: default
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 0 items / 577 deselected / -577 selected                                                                                                       

=== 577 deselected in 2.81s ===

No dice. Just confirming though, it does know that it's capture all of my tests in core/.

docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon core
=== test session starts ===
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: changed files: 0, skipping collection of 1830 files, environment: default
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 0 items / 577 deselected / -577 selected                                                                                                       

=== 577 deselected in 1.07s ===

And when I specify some other test to run, it runs them but gives me a negative 'selected' number:

docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon foo/tests/test_foo.py 
=== test session starts ===
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: changed files: 0, skipping collection of 1830 files, environment: default
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 2 items / 577 deselected / -575 selected                                                                                                       

foo/tests/test_foo.py ..   [100%]

=== 2 passed, 577 deselected in 4.32s ===

My pyproject.toml:

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "my_app.settings"
python_classes = "" # disable pytest-specific class discovery
python_files = ["*_tests.py", "test_*.py", "tests.py"]
python_functions = "" # disable pytest-specific function discovery

The only other information that might be useful: with a fresh .testmondata, when I try running core/tests/example and then run core/tests or core/, then it can find tests in core/tests/foo or core/bar/. It's unclear to me whether this only happens when the top-level directory changes, or whether it's some other variable (number of tests?).

[christianbundy]

Also maybe useful: it looks like newly created files are collected just fine.

docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon core
================================================================== test session starts ===================================================================
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: new DB, environment: default
We'd like to hear from testmon users! 🙏🙏 go to https://testmon.org/survey to leave feedback ✅❌
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 587 items                                                                                                                                      

[redacted]

=== 587 passed in 125.68s (0:02:05) ===
docker@834556fca415:~/code$ cp foo/tests/test_foo.py foo/tests/test_another.py
docker@834556fca415:~/code$ pytest -Wignore --no-migrations --reuse-db --testmon .
=== test session starts ===
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: my_app.settings (from env)
testmon: changed files: 0, skipping collection of 1833 files, environment: default
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collected 2 items / 587 deselected / -585 selected                                                                                                       

foo/tests/test_another.py ..                                                                                                             [100%]

=== 2 passed, 587 deselected in 7.15s ===

[pokopt]

HI @christianbundy and thanks for detailed information. Nevertheless I'm still not able to reproduce the problem or understand what can be wrong. Are you sure there is no way that tests from 'foo' directory were executed during the first pytest run? Can't there be some kind of dependency or import that makes files in 'foo' directory part of the execution of tests in 'core'? Would you mind to attach .testmondata database after first run of tests to get better overview what data testmon collected? Thanks in advance.

[christianbundy]

Are you sure there is no way that tests from 'foo' directory were executed during the first pytest run?

I'm reasonably confident that the tests aren't running, since they take a very long time.

Can't there be some kind of dependency or import that makes files in 'foo' directory part of the execution of tests in 'core'?

That's an interesting idea -- we do use something like this for dynamic imports, which could be somehow importing the files in foo/, but would certainly not be running the tests.

Would you mind to attach .testmondata database after first run of tests to get better overview what data testmon collected?

I don't think that I'll be able to do this, since it's a work project, but I'm happy to answer questions or run queries on the database. I can confirm that when running tests on core/, the testmondb doesn't reference any files from foo in any tables.

[tarpas]

I made a minimal example which reproduces an issue like this. There has to be a dependency of bar on foo (https://github.com/axdel/testmon-issue204/blob/3015de708c272a48b567d8e08518e6903752bdf4/bar/tests.py#L9) and the testmon database has to be created by selecting bar/tests. Then the tests in foo are not picked up even on subsequent runs.

It's not easy to fix. For now I would kindly ask you just to workaround it and create the database including all files first.

(.venv) ➜  testmon-issue204 git:(main) rm .testmondata 
(.venv) ➜  testmon-issue204 git:(main) pytest -v --testmon bar/tests.py
================================ test session starts ================================
platform darwin -- Python 3.11.0, pytest-7.2.1, pluggy-1.0.0 -- /Users/tibor/tmonworkspace/testmon-issue204/.venv/bin/python3.11
cachedir: .pytest_cache
testmon: new DB, environment: default
We'd like to hear from testmon users! 🙏🙏 go to https://testmon.org/survey to leave feedback ✅❌
django: settings: my_app.settings (from ini)
rootdir: /Users/tibor/tmonworkspace/testmon-issue204, configfile: pytest.ini
plugins: testmon-1.4.5, django-4.5.2
collected 1 item                                                                    

bar/tests.py::BarTestCase::test_bar_1 PASSED                                  [100%]

================================= 1 passed in 0.18s =================================
(.venv) ➜  testmon-issue204 git:(main) pytest -v --testmon             
================================ test session starts ================================
platform darwin -- Python 3.11.0, pytest-7.2.1, pluggy-1.0.0 -- /Users/tibor/tmonworkspace/testmon-issue204/.venv/bin/python3.11
cachedir: .pytest_cache
testmon: changed files: 0, skipping collection of 4 files, environment: default
django: settings: my_app.settings (from ini)
rootdir: /Users/tibor/tmonworkspace/testmon-issue204, configfile: pytest.ini
plugins: testmon-1.4.5, django-4.5.2
collected 0 items / 1 deselected / -1 selected                                      

=============================== 1 deselected in 0.01s ===============================
(.venv) ➜  testmon-issue204 git:(main) pytest -v                       
================================ test session starts ================================
platform darwin -- Python 3.11.0, pytest-7.2.1, pluggy-1.0.0 -- /Users/tibor/tmonworkspace/testmon-issue204/.venv/bin/python3.11
cachedir: .pytest_cache
django: settings: my_app.settings (from ini)
rootdir: /Users/tibor/tmonworkspace/testmon-issue204, configfile: pytest.ini
plugins: testmon-1.4.5, django-4.5.2
collected 3 items                                                                   

bar/tests.py::BarTestCase::test_bar_1 PASSED                                  [ 33%]
foo/tests.py::FooTestCase::test_foo_1 PASSED                                  [ 66%]
foo/tests.py::bait PASSED                                                     [100%]

================================= 3 passed in 0.09s =================================