Using system-packages leads to module import error

ReubenVandezande commented 11 months ago

I'm not able to import an installed dependency when running tests with system-packages. I have a ~convoluted setup with:

A system python installation with site-packages (global, maintained by others)
A hatch installation in a venv (local, maintained by me) which points to the system python (include-system-site-packages = true in the pyvenv.cfg)

For example, I have two projects:

hatch new "Hatch Demo1"
hatch new "Hatch Demo2"

With an extra dependency not installed in the system python (python==3.7.3):

[tool.hatch.envs.default]
dependencies = [
  "coverage[toml]>=6.5",
  "pytest",
  "impunity" # your package of choice
]
dev-mode = true

I use dev-mode in both and system-packages in hatch-demo1 but not hatch-demo2. Running tests (hatch run test) results in an import error (ModuleNotFoundError: No module named 'impunity') for hatch-demo1 but not `hatch-demo2:

tests/test_import.py

def test_import_pytest():
    import impunity

To make this more complicated, I can import impunity no problem using hatch run python to start an interpreter and importing impunity.

I've noticed that my sys.path is different between the failing/passing scenarious where the failing test environment has the following for sys.path (/opt/software/python is the global install):

>>> import sys
>>> sys.path
['~/software/git_repos/hatch-demo/hatch-demo1', '/home/shelf1/Software/python/3.7/bin', '/opt/Software/python/3.7/lib/python37.zip', '/opt/Software/python/3.7/lib/python3.7', '/opt/Software/python/3.7/lib/python3.7/lib-dynload', '~/.local/lib/python3.7/site-packages', '/opt/Software/python/3.7/lib/python3.7/site-packages']

But the environment that successfully imports impunity has the following, noting the path to the virtual environment.

>>> import sys
>>> sys.path
['', '/opt/Software/python/3.7/lib/python37.zip', '/opt/Software/python/3.7/lib/python3.7', '/opt/Software/python/3.7/lib/python3.7/lib-dynload', '~/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/hatch-demo1/lib/python3.7/site-packages', '~/software/git_repos/hatch-demo/hatch-demo1', '~/.local/lib/python3.7/site-packages', '/opt/Software/python/3.7/lib/python3.7/site-packages']

ofek commented 11 months ago

Does the system-packages option work for you? https://hatch.pypa.io/latest/plugins/environment/virtual/#options

ReubenVandezande commented 11 months ago

@ofek That is the offending option, but I guess I can make that more clear. If I set system-packages to false, I'm able to correctly import the impunity (ie. non-system package), but when I specify system-packages = true, my tests fail (hatch run test).

I've included a parital pyproject.toml below with my default environment setup:

[tool.hatch.envs.default]
dependencies = [
  "coverage[toml]>=6.5",
  "pytest",
  "impunity"
]
system-packages = false # setting to true causes import issues while running tests

[tool.hatch.envs.default.scripts]
test = "pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
  "- coverage combine",
  "coverage report",
]
cov = [
  "test-cov",
  "cov-report",
]

ofek commented 11 months ago

Do you mind running hatch env prune and then testing with the code on the master branch? There may have been fixes recently.

ReubenVandezande commented 11 months ago

That solved one issue, but I'm now in another pickle since hatch dropped support for 3.7 (understandably so). I was hoping I could get away with pointing to the 3.7 binary on my system (using the python setting), but I've had similar issues as before. Pointing to 3.8 resulted in similar issues. I've listed my environment and steps to reproduce in python3.8 below.

Testing environment:

[tool.hatch.envs.test]
dependencies = [
  "coverage[toml]>=6.5",
  "pytest",
  "impunity"
]
python = "/opt/Software/python/3.8/bin/python"
system-packages = true

[tool.hatch.envs.test.scripts]
env = "python -c 'import sys;print(sys.path);import pytest;import impunity'"
test = "pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
  "- coverage combine",
  "coverage report",
]
cov = [
  "test-cov",
  "cov-report",
]

Steps to reproduce

>>> source "/path/to/hatch/bin/activate"
>>> hatch run prune
>>> hatch run test:env
[
  '', 
  '/opt/Software/python/3.8/lib/python38.zip', 
  '/opt/Software/python/3.8/lib/python3.8', 
  '/opt/Software/python/3.8/lib/python3.8/lib-dynload', 
  '/beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/test/lib/python3.8/site-packages',
  '/beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1', 
  '/opt/Software/python/3.8/lib/python3.8/site-packages'
]
>>> hatch run test:test
/bin/sh: pytest: command not found

ofek commented 11 months ago

python -m pytest

ReubenVandezande commented 11 months ago

>>> python -m pytest
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.3.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-4.0.0
collected 1 item                                                                                                                                                        

tests/test_import.py F                                                                                                                                            [100%]

=============================================================================== FAILURES ================================================================================
__________________________________________________________________________ test_import_pytest ___________________________________________________________________________

    def test_import_pytest():
>       import impunity
E       ModuleNotFoundError: No module named 'impunity'

tests/test_import.py:4: ModuleNotFoundError
======================================================================== short test summary info ========================================================================
FAILED tests/test_import.py::test_import_pytest - ModuleNotFoundError: No module named 'impunity'
=========================================================================== 1 failed in 0.18s ===========================================================================

ReubenVandezande commented 11 months ago

I added a matrix of python (3.7, 3.8, 3.11) and added just the binaries to my $PATH.

Running `hatch run test:test`

hatch run test:test
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
/bin/sh: pytest: command not found

Running `hatch run test:python -m pytest`

>>> hatch run test:python -m pytest
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.7.3, pytest-5.0.0, py-1.8.0, pluggy-0.12.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: arraydiff-0.3, astropy-header-0.1.2, remotedata-0.3.2, openfiles-0.5.0, doctestplus-0.11.1
collected 1 item                                                                                                                                                        

tests/test_import.py .                                                                                                                                            [100%]

=========================================================================== warnings summary ============================================================================
tests/test_import.py::test_import_pytest
tests/test_import.py::test_import_pytest
  /opt/Software/python/3.7/lib/python3.7/site-packages/pytest_remotedata/plugin.py:65: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(pytest.__version__) < LooseVersion("3.6"):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================= 1 passed, 2 warnings in 0.20 seconds ==================================================================
────────────────────────────────────────────────────────────────────────────── test.py3.8 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.8.16, pytest-7.3.1, pluggy-1.0.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-3.5.0
collected 1 item                                                                                                                                                        

tests/test_import.py .                                                                                                                                            [100%]

=========================================================================== 1 passed in 0.05s ===========================================================================
────────────────────────────────────────────────────────────────────────────── test.py3.11 ──────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.0.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
plugins: anyio-3.5.0
collected 1 item                                                                                                                                                        

tests/test_import.py .                                                                                                                                            [100%]

=========================================================================== 1 passed in 0.15s ===========================================================================

ReubenVandezande commented 11 months ago

Bonus round

Setting test = "python -m pytest {args:tests}" works, but leads to library issues since impunity is compiled. Those errors are on my side, so this error is expected.

>>> hatch run test:test
────────────────────────────────────────────────────────────────────────────── test.py3.7 ───────────────────────────────────────────────────────────────────────────────
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.7.9, pytest-7.4.3, pluggy-1.2.0
rootdir: /beegfs/users/rvandezande/software/git_repos/hatch-demo/hatch-demo1
collected 1 item                                                                                                                                                        

tests/test_import.py F                                                                                                                                            [100%]

=============================================================================== FAILURES ================================================================================
__________________________________________________________________________ test_import_pytest ___________________________________________________________________________

    def test_import_pytest():
>       import impunity
E       ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/test.py3.7/lib/python3.7/site-packages/impunity.cpython-37m-x86_64-linux-gnu.so)

tests/test_import.py:4: ImportError
======================================================================== short test summary info ========================================================================
FAILED tests/test_import.py::test_import_pytest - ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /beegfs/users/rvandezande/.local/share/hatch/env/virtual/hatch-demo1/jxt6XpzL/te...
=========================================================================== 1 failed in 0.43s ===========================================================================

ReubenVandezande commented 11 months ago

@ofek I have a workaround for whatever is causing these issues. I was able to import everything successfully by combining the above solution with an absolute path to the system python. This is slightly unfortunate as I won't be able to use the matrix inputs for testing.

Successful import with python3.7

[tool.hatch.envs.test]
dependencies = [
  "coverage[toml]>=6.5",
  "pytest",
  "impunity"
]
python = "/opt/Software/python/3.7/bin/python3.7"
system-packages = true

[tool.hatch.envs.test.scripts]
env = "python -c 'import sys;print(sys.path)'"
test = "python3.7 -m pytest {args:tests}"
test-cov = "coverage run -m pytest {args:tests}"
cov-report = [
  "- coverage combine",
  "coverage report",
]
cov = [
  "test-cov",
  "cov-report",
]

Error requiring full python name

Note that setting the test script to just python -m pytest {args:test} resulted in an error. It looks like python was stripped from the command.

cmd [1] | 3.7 -m pytest tests
/bin/sh: 3.7: command not found

pypa / hatch