`--pyargs` test discovery fails in frozen application

[bersbersbers]

I am having trouble with --pyargs test discovery in a frozen application. I am using Windows 10 22H2, pytest 7.4.2, pyinstaller 6.1.0.

Minimal example

app.py

import sys

import pytest

import folder.test_file

sys.exit(pytest.main(sys.argv[1:]))

folder\test_file.py

print("Hello world!")

def test_pass():
    pass

run.bat

@echo off

rem Run before deployment - import and test discovery work
python -c "import folder.test_file"
pytest --pyargs folder

rem Deploy
if exist dist\app.exe del dist\app.exe
pyinstaller --log-level=DEPRECATION --collect-submodules folder --hidden-import folder --hidden-import folder.test_file --onefile app.py

rem Rename folder
ren folder folder.bak

rem Run after deployment - import works, but test discovery does not
dist\app.exe --pyargs folder

rem Rename folder back
ren folder.bak folder

Output:

Hello world!
======================================================================================= test session starts ======================================================================================= 
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
PySide6 6.5.3 -- Qt runtime 6.5.3 -- Qt compiled 6.5.3
rootdir: C:\Code\project
plugins: pylama-8.4.1, qt-4.2.0
collected 1 item

folder\test_file.py .                                                                                                                                                                        [100%] 

======================================================================================== 1 passed in 0.05s ======================================================================================== 
Hello world!
======================================================================================= test session starts ======================================================================================= 
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: C:\Code\project
collected 0 items

====================================================================================== no tests ran in 0.00s ====================================================================================== 
ERROR: module or package not found: folder (missing __init__.py?)

Since import works, I guess that this is not primarily a pyinstaller issue, but do let me know if you think it is :)

[RonnyPfannschmidt]

this is most likely related to finding the import path

whats the result if importlib searching the module in the frozen executable

ps https://github.com/pytest-dev/pytest/blob/cdddd6d69533c07092c64445a0d42d1fc75c8299/src/_pytest/main.py#L849-L913 implements the logic for the case

[bersbersbers]

whats the result if importlib searching the module in the frozen executable

I added

module_name = "folder"
spec = importlib.util.find_spec(module_name)
print(spec)
print(spec.origin is None)

to my code, and this is what I get:

ModuleSpec(name='folder', loader=<_frozen_importlib_external.NamespaceLoader object at 0x000002329F3B1A90>, submodule_search_locations=_NamespacePath(['C:\\Users\\bers\\AppData\\Local\\Temp\\_MEI121682\\folder']))
True

So spec.origin is None.

[bersbersbers]

By the way, what works is (also) including test files as data, using --add-data folder/test_file.py;folder and then calling something like sys.exit(pytest.main(Path(__file__).parent / "folder")). But that looks like a nasty workaround to me, since PyInstaller already needs to include test_file.py as code to follow all dependencies, and then needs to include it another time as data.

[RonnyPfannschmidt]

my understanding is, that pytest currently has no collector able to load test modules from frozen pyinstaller stuff

[bersbersbers]

pytest currently has no collector able to load test modules from frozen pyinstaller stuff

Having looked at the collection a bit now, I think I agree. Everything looks pretty much Path-based, which I understand "pyinstaller stuff" is not.

So at least I didn't file an issue that has an RTFM-kind of answer 😉