search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.93k stars 2.66k forks source link

File execution order different when using class/method notation in CLI #11303

Closed ee-4-me closed 1 year ago

ee-4-me commented 1 year ago

Pytest

7.4.0

OS

Windows 10 Enterprise

Overview

Issue where test files are not run in order when using test class/method notation in CLI. This issue arose when creating a test setup file to make test setup easier (as opposed to other methods).

Example Setup

Folder structure: 

|-- src
    |-- add.py
    |-- opp.py
|-- test
    |-- test_opp.py

add.py: 

def add_2(a, b):
    return a + b

opp.py

from add import add_2

class OppClass:
    def opp_add_2(self, a, b):
        return add_2(a, b)

test_opp.py

from opp import OppClass

class Test_Opp:
    def test_add_2(self):
        oppClass = OppClass()

        assert 2 == oppClass.opp_add_2(1, 1)
        assert 3 == oppClass.opp_add_2(2, 1)

When executing python -m pytest ./test/test_opp.py, I get the following:

PS C:\Users\User\Documents\pytesterror> python -m pytest ./test/test_opp.py
========================================================= test session starts ==========================================================
platform win32 -- Python 3.10.10, pytest-7.4.0, pluggy-1.2.0
PyQt5 5.15.9 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2     
rootdir: C:\Users\User\Documents\pytesterror
plugins: cov-4.1.0, qt-4.2.0
collected 0 items / 1 error

================================================================ ERRORS ================================================================
__________________________________________________ ERROR collecting test/test_opp.py ___________________________________________________
ImportError while importing test module 'C:\Users\User\Documents\pytesterror\test\test_opp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\..\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test\test_opp.py:1: in <module>
    from opp import OppClass
E   ModuleNotFoundError: No module named 'opp'
======================================================= short test summary info ========================================================
ERROR test/test_opp.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================== 1 error in 0.09s ===========================================================

Okay, makes sense, there is an error with the import paths.
To simplify adding the import path to sys.path for my test file (in actuality, dozens of test files), I create the following test_setup.py: 

|-- src
    |-- add.py
    |-- opp.py
|-- test
    |-- test_opp.py
    |-- test_setup.py NEW

test_setup.py

import sys

# Add src to sys.path so all test files do not need to add it in manually
sys.path.insert(0, 'src')

Now, when running python -m pytest ./test/test_setup.py ./test/test_opp.py, the test runs and works:

PS C:\Users\User\Documents\pytesterror> python -m pytest ./test/test_setup.py ./test/test_opp.py
========================================================= test session starts ==========================================================
platform win32 -- Python 3.10.10, pytest-7.4.0, pluggy-1.2.0
PyQt5 5.15.9 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: C:\Users\User\Documents\pytesterror
plugins: cov-4.1.0, qt-4.2.0
collected 1 item

test\test_opp.py .                                                                                                                [100%]

========================================================== 1 passed in 0.01s ===========================================================

Wonderful, this is working great, I am taking advantage of naming my setup file test_ so that it runs in pytest, and I can add the import before all of the other test files run (among any other things, without refactoring my dozens of test files).

Bug

The bug, however, is if I want to run a test "class" or test "method" with this setup.
For example, running python -m pytest ./test/test_setup.py ./test/test_opp.py::Test_Opp 

PS C:\Users\User\Documents\pytesterror> python -m pytest ./test/test_setup.py ./test/test_opp.py::Test_Opp
========================================================= test session starts ==========================================================
platform win32 -- Python 3.10.10, pytest-7.4.0, pluggy-1.2.0
PyQt5 5.15.9 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: C:\Users\User\Documents\pytesterror
plugins: cov-4.1.0, qt-4.2.0
collected 0 items / 1 error

================================================================ ERRORS ================================================================ 
__________________________________________________ ERROR collecting test/test_opp.py ___________________________________________________ 
ImportError while importing test module 'C:\Users\User\Documents\pytesterror\test\test_opp.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\..\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test\test_opp.py:1: in <module>
    from opp import OppClass
E   ModuleNotFoundError: No module named 'opp'
======================================================= short test summary info ======================================================== 
ERROR test/test_opp.py
=========================================================== 1 error in 0.07s =========================================================== 
ERROR: not found: C:\Users\User\Documents\pytesterror\test\test_opp.py::Test_Opp
(no name 'C:\\Users\\User\\Documents\\pytesterror\\test\\test_opp.py::Test_Opp' in any of [<Module test/test_opp.py>])

It appears that by adding the class option, ::Test_Opp, the test_opp.py file was run (or pre-run / compiled / pytest magic-ed) before test_setup.py

This appears to be a bug, that the order of file execution changes when adding a class option, ::, to a test in the CLI args.
I would think that python -m pytest ./test/test_setup.py ./test/test_opp.py::Test_Opp should run ./test/test_setup.py before ./test/test_opp.py::Test_Opp.

This particular example may not make the strongest case for this change (as there are other ways to solve the path import issue for the test files), however, I think it points out an unexpected outcome with the test order in the CLI.

Workaround

On a side note, I can workaround this issue if I wrap the functionality in test_setup with a "test class":
test_setup.py 

import sys

# Add src to sys.path so all test files do not need to add it in
class Test_Setup: # Wrap in class
    sys.path.insert(0, 'src')

And then run the tests using both class options python -m pytest ./test/test_setup.py::Test_Setup ./test/test_opp.py::Test_Opp:

PS C:\Users\User\Documents\pytesterror> python -m pytest ./test/test_setup.py::Test_Setup ./test/test_opp.py::Test_Opp
========================================================= test session starts ==========================================================
platform win32 -- Python 3.10.10, pytest-7.4.0, pluggy-1.2.0
PyQt5 5.15.9 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: C:\Users\User\Documents\pytesterror
plugins: cov-4.1.0, qt-4.2.0
collected 1 item

test\test_opp.py .                                                                                                                [100%] 

========================================================== 1 passed in 0.03s ===========================================================

The tests execute as expected.

Appendix

pip list:

Package                   Version
------------------------- ---------
altgraph                  0.17.3
annotated-types           0.5.0
azure-core                1.28.0
azure-storage-blob        12.17.0
blinker                   1.6.2
can                       0.0.0
canlib                    1.23.804
certifi                   2023.7.22
cffi                      1.15.1
charset-normalizer        3.2.0
click                     8.1.6
colorama                  0.4.6
coverage                  7.2.7
cryptography              38.0.3
DateTime                  5.2
defusedxml                0.7.1
docopt                    0.6.2
exceptiongroup            1.1.2
Flask                     2.3.2
gcovr                     6.0
idna                      3.4
iniconfig                 2.0.0
isodate                   0.6.1
itsdangerous              2.1.2
Jinja2                    3.1.2
joblib                    1.3.1
kvaser                    0.1.7
lxml                      4.9.3
markdown-it-py            3.0.0
MarkupSafe                2.1.3
mdurl                     0.1.2
mock                      5.0.2
networkx                  3.1
numpy                     1.25.1
packaging                 23.1
pandas                    2.0.3
patsy                     0.5.3
pefile                    2023.2.7
Pillow                    10.0.0
pip                       23.2.1
pipreqs                   0.4.13
plotly                    5.15.0
pluggy                    1.2.0
psutil                    5.9.4
pycparser                 2.21
pydantic                  2.1.1
pydantic_core             2.4.0
pydot                     1.4.2
Pygments                  2.15.1
pyinstaller               5.13.0
pyinstaller-hooks-contrib 2023.6
pyparsing                 3.1.0
PyQt5                     5.15.9
PyQt5-Qt5                 5.15.2
PyQt5-sip                 12.12.0
pyqtgraph                 0.12.4
pyserial                  3.5
pytest                    7.4.0
pytest-cov                4.1.0
pytest-qt                 4.2.0
python-can                4.0.0
python-dateutil           2.8.2
pytz                      2023.3
pywin32                   306
pywin32-ctypes            0.2.2
requests                  2.31.0
rich                      13.4.2
RP1210                    0.0.26
scikit-learn              1.3.0
scipy                     1.11.1
setuptools                65.5.0
six                       1.16.0
tenacity                  8.2.2
threadpoolctl             3.2.0
toml                      0.10.2
tomli                     2.0.1
typing_extensions         4.7.1
tzdata                    2023.3
urllib3                   2.0.4
Werkzeug                  2.3.6
windows-curses            2.3.1
wrapt                     1.15.0
yarg                      0.1.9
zope.interface            6.0
bluetech commented 1 year ago

pytest does not guarantee the order of imports, doing so will be too restrictive. You mentioned that you've found other ways to solve your problem (I can recommend the pythonpath option), instead of relying on import-time side-effects, which is the way to go.

Just in case you're curious why the behavior you're seeing happens, it's because pytest needs to look into the file to figure out what Test_Opp means in order to collect it, and this happens before importing the files looking for tests. But this is all subject to change.