tool.hatch.envs.default.python version is not being used in hatch test

[cdwilson]

I assumed that running hatch test would run the tests using the default environment, and that by setting tool.hatch.envs.default.python = "3.8", hatch would run the tests using Python 3.8.

What I'm seeing is that hatch test runs against the python version used to install hatch (via pipx), ignoring the python = "3.8" setting in the default environment.

How to reproduce:

1. Set the following in pyproject.toml:

[tool.hatch.envs.default]
python = "3.8"

2. Run hatch test (no hatch shell activated)

The output shows that Python 3.12.4 is being used:

❯ hatch test
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
...

However, if I run hatch shell, the correct version of python is activated in the shell:

❯ hatch shell
❯ python --version
Python 3.8.19

But, if I run hatch test while the shell is active, the hatch test command uses the incorrect version of python:

❯ hatch shell
❯ python --version
Python 3.8.19
❯ hatch test
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
...

[cdwilson]

I assumed that running hatch test would run the tests using the default environment

This assumption was based on this line in the testing tutorial:

If no environment options are selected, the test command will only run tests in the first defined environment that either already exists or is compatible.

I assumed that since I don't have any environments defined in pyproject.toml, the "first defined environment" would be the default one.

However, I just found the testing configuration docs page and realized that there appears to be an internal hatch-test environment. Is this the default environment that is used when running hatch test?

I tried adding the following:

[tool.hatch.envs.hatch-test]
python = "3.8"

However, this produced the same results:

❯ hatch test
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0

[cdwilson]

I think I figured out what's going on here.

The hatch-test environment defines a matrix including multiple python versions:

[[tool.hatch.envs.hatch-test.matrix]]
python = ["3.12", "3.11", "3.10", "3.9", "3.8"]

As a result, this generates the following environments:

hatch-test.py3.12
hatch-test.py3.11
hatch-test.py3.10
hatch-test.py3.9
hatch-test.py3.8

So, running hatch test is actually running against the hatch-test.py3.12 environment since hatch is installed on my machine using Python 3.12 (via pipx).

For example, if I change the matrix in hatch-test so that 3.8 is first in the list of python versions (and remove the 3.12 "compatible" version), hatch test will use 3.8.

[[tool.hatch.envs.hatch-test.matrix]]
python = ["3.8", "3.9"]

❯ hatch test
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.8.19, pytest-8.2.2, pluggy-1.5.0

[doolio]

Thanks for your report and subsequent investigations. I too observed the same behaviour today and couldn't figure out why. However, based on your findings wouldn't one assume it tests against all python versions available? Otherwise what purpose does defining such a matrix have? It seems it tests against the first version found in that matrix list. For example, my system python is 3.9.2 (which is what was used to install hatch. I've also installed version 3.8.20 with hatch but with the default matrix it only seems to test against the system version and not the 3.8 which like you I set as the default environment version. or at least that is all you see in the output. I would have expected to see a test output for each python version available.

@ofek what are we missing here? Thanks.

Edit: Never mind. Following the above link to issue#1624 led me to the appropriate section in the docs. Sorry for the noise.