Use the same environment for both pytest collect/run phases

[zed]

Specific Maelstrom Program? maelstrom-pytest

Problem Statement Current behavior: maelstorm-pytest runs pytest in the local environment to collect tests, then tests are run inside containers e.g., created using test-requirements.txt It may create subtle differences and corresponding bugs.

Proposed Solution

Run both collect/run test phases in the environments created the same way (both run inside containers).

Additional Context

It is also more convenient to replace (current behavior):

git checkout "$TEST_BRANCH"  # get source
uv venv .venv
source .venv/bin/activate
uv pip install -r test-requirements.txt
maelstrom-pytest  # run unit tests

with

git checkout "$TEST_BRANCH"  # get source
maelstrom-pytest  # run unit tests

[bobbobbio]

Thank you for the issue. I agree that this seems like a good idea.

Collecting the tests by running some job is unlike the way our other two test runners work, but I think for Python it makes sense.

We will need a way to specify what the environment looks like when listing tests, I think once we finish our named container specification, we can maybe require some specific named container exist which specifies the environment when listing.

[zed]

No special environment is necessary that is the point of the issue: --collect-only and test pytest invocations should be run in as close environment as possible. It may affect which tests are collected, and avoiding relying on local dev environment is a good idea for tests in general.

[bobbobbio]

I agree with what you are saying.

Our test metadata configuration allows us to configure the environment (by environment I mean how the container looks, both the files available but also any OS or devices likes networking etc) in which tests are run and it allows it to be different per test. This includes how the container is set-up but also what files we include in the container.

For pip packages currently we have some hard-coded environment we use to run the pip install and collect the resulting files. These are then included automatically in the test environment. We see it as undesirable that this environment isn't in some way configurable and are planning in a future release to allow the user to specify this environment too.

Of course for both of these we are planning on having sane defaults so if you don't need anything special it won't need to be configured.

For the step of running pytest with --collect-only I was trying to say that we would ideally let this environment be configurable also with a good default. There isn't some existing described environment we could use, the one for tests is potentially specific per test.

I can think of many reasons why both the pip install or pytest --collect steps would want to be in some way modified. Perhaps the user wants to use some base container which is a different flavor of Linux OS than whatever we picked. Maybe there are extra files that the user wants to have access to as part of some plugin, or maybe the user want to rearrange the layout of the container in some way.