PyO3 error

[LukasJerabek]

Hello,

I am currently debugging weird issue that happens to us only when using --cov arguments to pytest and only during running tests (not in production with the same code).

Reproducer

I was able to narrow it down for this little setup: create this structure:

.
├── folder
│   ├── __init__.py
│   └── some_module.py
└── test_one.py

Now, some_module.py is empty. folder/__init__.py has this content:

`from cryptography.hazmat.primitives import hashes`

test_one.py has this content:

`from cryptography.hazmat.primitives import hashes`

When you run this command (note that in my case the whole directory structure is in /home/vagrant/neto directory):

`PYTHONPATH=$PYTHONPATH:/home/vagrant/neto pytest /home/vagrant/neto/test_one.py --cov=folder.some_module`

you get

ImportError while importing test module '/home/vagrant/neto/test_one.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../.pyenv/versions/3.8.18/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_one.py:2: in <module>
    from cryptography.hazmat.primitives import hashes
../venv/pack/lib/python3.8/site-packages/cryptography/hazmat/primitives/hashes.py:10: in <module>
    from cryptography.hazmat.bindings._rust import openssl as rust_openssl
E   ImportError: PyO3 modules may only be initialized once per interpreter process

explanation

Reasons why I think (though not sure) the problem is in python-cov:

• you can switch from cryptography.hazmat.primitives import hashes for import jsonschema and you get the into the same error.
• we dont get any errors in production with the other imported packages.
• you remove the --cov attribute and it runs without errors.
• that importError sounds like that it might be caused by the way --cov path is handled internally?

versions

• python 3.8.18
• pytest 7.4.2
• pytest-cov 4.1.0
• jsonschema 4.19.1, problem doesnt appear when downgrading to 4.17.3
• cryptography 41.0.4, problem doesnt appear when downgrading to 40.0.2

[pga2rn]

I also encounter this issue recently while introducing pydantic into my code base and local pytest is broken due to the same ImportError: PyO3 modules may only be initialized once per interpreter process .

I am using the python 3.8.10 with pytest 7.12, pytest-cov 3.0.0, pydantic 2.5.1.

---

First I try to find out what is going on on pyo3 side. pydantic-core uses rust extension with pyo3, and pyo3 recently(1~2 months ago?) introduces a check to explicitly rejecting multiple module initializing in the same process, as this behavior is not supported.

---

Then I try to find out why pyo3 module is initialized multiple times when running pytest.

Since pytest-cov actually use coverage.py under the hook, I read through coverage.py's document and find something interesting:

in Specifying source files :

You can specify source to measure with the --source command-line switch, or the [run] source configuration value.

Modules named as sources may be imported twice, once by coverage.py to find their location, then again by your own code or test suite. Usually this isn’t a problem, but could cause trouble if a module has side-effects at import time.

So the problem is caused by specifying package to be measured by coverage.py "sourcing" them, this results in module being initialized twice(happens inside the same interpreter process starts by coverage.py), and I do specify which package to measure via --cov= parameter for pytest-cov.

Not 100% sure how pytest-cov translates --cov= parameters into coverage.py's corresponding configs, but I guess --cov=<package> is translated into [run] source = [<package>, ] if the <package> param is in module path format(pkga.pkgb)?

---

By checking the Specifying source files further, actually coverage.py provides us another method to specify packages, which is by packages' file path.

So I try to run the test directly with coverage(coverage run -m pytest), and specify packages to measured by specifying the file paths([run] include = [<files_to_be_measured>, ...]), no-more ImportError from pyo3 and everything works again!