testmon does not respect test order

[emavgl]

Hello again, I have a problem related to dependent tests

# gold_user.py

import pathlib

class GoldUser:
    def __init__(self, location) -> None:
        self.file_path = pathlib.Path(location, "file.json")

    def load_table(self):
        with open(self.file_path, "w+") as f:
            json.dump({"test": 1}, f)

    def do_something(self):
        with open(self.file_path) as f:
            result = json.load(f)
            return result["test"]

# tests/gold_test_user.py

import pytest
import json
import pathlib
from gold_user import GoldUser

class TestGoldUser:
    @pytest.fixture(scope="module")
    def gold_pipeline(self, silver_country_mapping_table_location):
        return GoldUser(silver_country_mapping_table_location)

    def test_load_table(self, gold_pipeline):
        gold_pipeline.load_table()

    def test_gold_table_after_batch_1(self, gold_pipeline):
        assert 1 == gold_pipeline.do_something()

In this case, there is a dependency between tests. test_load_table writes a file, that is then read by the method test_gold_table_after_batch_1 (I know it is best practice having independent tests, but I am trying to simply a more complex use case).

We usually run

python -m pytest -n 4 tests --dist loadfile

that respect the execution order of the tests in the same file. The first run:

python -m pytest -n 4 tests --dist loadfile --testmon        
====================================================== test session starts ======================================================
platform darwin -- Python 3.7.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
testmon: new DB, environment: default
We'd like to hear from testmon users! 🙏🙏 go to https://testmon.org/survey to leave feedback ✅❌

rootdir: /Users/emv/repos/test-testmon2
plugins: anyio-2.2.0, xdist-2.3.0, testmon-1.4.5, html-3.1.1, html-reporter-0.2.6, cov-2.11.1, doubles-1.5.3, spark-0.6.0, mock-3.6.1, metadata-1.11.0, forked-1.3.0
gw0 [5] / gw1 [5] / gw2 [5] / gw3 [5]
.....                                                                                                                     [100%]
======================================================= 5 passed in 1.43s ======================================================

Assuming now we change both method in the class:

# gold_user.py

import pathlib

class GoldUser:
    def __init__(self, location) -> None:
        self.file_path = pathlib.Path(location, "file.json")

    def load_table(self):
        with open(self.file_path, "w+") as f:
            json.dump({"test": 2}, f) # 2 instead of 1

    def do_something(self):
        with open(self.file_path) as f:
            result = json.load(f)
            return result["test"] + 1  - 1 # summing and removing 1

I expect the previous tests to be selected and run in the following order:

• test_load_table
• test_gold_table_after_batch_1

However, if I run

python -m pytest -n 4 tests --dist loadfile --testmon-nocollect --collect-only -q
tests/gold_user/test_gold_user.py::TestGoldUser::test_gold_table_after_batch_1
tests/gold_user/test_gold_user.py::TestGoldUser::test_load_table

-1/2 tests collected (3 deselected) in 0.01s

The order of the two tests is inverted. Resulting in my testsuite to fail because the data were not loaded:

python -m pytest -n 4 tests --dist loadfile --testmon        
====================================================== test session starts ======================================================
platform darwin -- Python 3.7.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
testmon: new DB, environment: default
We'd like to hear from testmon users! 🙏🙏 go to https://testmon.org/survey to leave feedback ✅❌

rootdir: /Users/emv/repos/test-testmon2
plugins: anyio-2.2.0, xdist-2.3.0, testmon-1.4.5, html-3.1.1, html-reporter-0.2.6, cov-2.11.1, doubles-1.5.3, spark-0.6.0, mock-3.6.1, metadata-1.11.0, forked-1.3.0
gw0 [5] / gw1 [5] / gw2 [5] / gw3 [5]
.....                                                                                                                     [100%]
======================================================= 5 passed in 1.43s =======================================================
(base)  emv@ofc-02564-m  ~/repos/test-testmon2  python -m pytest -n 4 tests --dist loadfile --testmon-nocollect --collect-only -q
tests/gold_user/test_gold_user.py::TestGoldUser::test_gold_table_after_batch_1
tests/gold_user/test_gold_user.py::TestGoldUser::test_load_table

-1/2 tests collected (3 deselected) in 0.01s
(base)  emv@ofc-02564-m  ~/repos/test-testmon2  python -m pytest -n 4 tests --dist loadfile --testmon
====================================================== test session starts ======================================================
platform darwin -- Python 3.7.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
testmon: changed files: tests/gold_user/test_gold_user.py, skipping collection of 4 files, environment: default

rootdir: /Users/emv/repos/test-testmon2
plugins: anyio-2.2.0, xdist-2.3.0, testmon-1.4.5, html-3.1.1, html-reporter-0.2.6, cov-2.11.1, doubles-1.5.3, spark-0.6.0, mock-3.6.1, metadata-1.11.0, forked-1.3.0
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
F.                                                                                                                        [100%]
=========================================================== FAILURES ============================================================
__________________________________________ TestGoldUser.test_gold_table_after_batch_1 ___________________________________________
[gw0] darwin -- Python 3.7.6 /opt/anaconda3/bin/python

self = <test_gold_user.TestGoldUser object at 0x7f7908469f90>, gold_pipeline = <gold_user.GoldUser object at 0x7f7908481550>

    def test_gold_table_after_batch_1(self, gold_pipeline):
>       assert 1 == gold_pipeline.do_something()

tests/gold_user/test_gold_user.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <gold_user.GoldUser object at 0x7f7908481550>

    def do_something(self):
>       with open(self.file_path) as f:
E       FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/y_/hknpfl0d6xbfg89rp5shx_kc0000gn/T/pytest-of-emv/pytest-78/popen-gw0/silver_country_mapping0/file.json'

gold_user.py:14: FileNotFoundError
--------------------------------------------------- Captured stdout teardown ----------------------------------------------------
F
==================================================== short test summary info ====================================================
FAILED tests/gold_user/test_gold_user.py::TestGoldUser::test_gold_table_after_batch_1 - FileNotFoundError: [Errno 2] No such f...
================================================== 1 failed, 1 passed in 1.49s ==================================================

I am thinking, it would be also helpful, to have a flag in testmon, to force the run of every tests in the same file. Because if I have tests with dependencies, like in this case, and just test_gold_table_after_batch_1 is selected to run, it will fail because it needs test_load_table test to be executed first.

Hopefully I am not missing any step. Thank you again for your help and your work!

[emavgl]

Could https://github.com/tarpas/pytest-testmon/blob/main/testmon/pytest_testmon.py#L408 be the cause of the problem? Is this sorting actually necessary? @tarpas If not necessary, I would suggest to remove it, since there could be test-suites that depends on test-ordering.

[tarpas]

I'm sorry but reordering tests is main feature of testmon and will become more prominent in the next release. In many projects we'll not be able to conclusively filter tests, so testmon will strive to come to failure quickly and end the execution after the first or a couple of failures.

I'm of two minds about the tradeoffs of some kind formal test dependency specification (e.g. via pyteset-dependency). Feel free to open a feature request about a support for that and also what rationale do you have to go agains first principle of automated tests (independence).