Testmon very sensitive towards library changes

[dgot]

Background

I have found pytest-testmon to be very sensitive towards the slightest changes for the packages in the environment it is installed. This makes very difficult to use pytest-testmon in practice when sharing a .testmondata file between our developers.

Use Case:

A developer is trying out a new library foo and have written a wrapper foo_bar.py for this. The developer only wants to run test_foo_bar.py for foo_bar.py, however the entire test suite is run since foo was installed.

Example Solution

Flag for disregarding library changes

• I am happy to contribute with a solution, if it is considered possible.
• If library needs funding for a solution, this is an option as well.

[emavgl]

Could this also be true for every changes in the requirements.txt?

[tarpas]

@dgot @emavgl Hello. I just pushed a beta version (install with pip install --pre pytest-testmon) which disregards changes of dependencies versions in the patch version number (after 2nd dot). Could you please try!?

[tarpas]

Please let me knnow if disregarding version after 2nd dot is practical enough.

[emavgl]

Hello! Unfortunately I don't have much time these days to try this out. Also, I am not currently using testmon at work. So, from my side, please feel free to merge the change.

[jacksongoode]

Sorry to be back one again with an issue, but the latest version is throwing. testmon: The packages installed in your Python environment have been changed. All tests have to be re-executed. in some cases and not skipping tests that haven't been changed in some case.

Our process is:

• Retrieve the last testmondata(-shm/wal) produced from running the last tests
• Use setup.py to read package & version from lock (we have a Pipfile => Pipfile.lock)
• Build wheel with setup.py
• Send wheel to a Databricks Spark cluster (a new Linux environment)
• Install the wheel
• Find that when tests are triggered with .testmondata, packages are wrong, or tests are not skipped.

My guess is just that reinstalling all the dependencies on a remote environment ends up installing packages with the same version but different hashes?

[tarpas]

Hi Jackson, the code which retrieves current packages is here https://github.com/tarpas/pytest-testmon/blob/72c70c79aa85095df45bc201adb13bc8e7f7ec96/testmon/common.py#L36 Please create a test which uses that to get and print your environment. Then you'll see what changes. It's quite likely that your own package/packages changes with every version. You can than use the undocumented testmon_ignore_dependencies configuration option to eliminate the unwanted package. (https://github.com/tarpas/pytest-testmon/blob/72c70c79aa85095df45bc201adb13bc8e7f7ec96/testmon/pytest_testmon.py#L140C5-L140C71)

As for more intuitive and documented solutions. We'll drop the sensitivity in the future quite a bit more

[jacksongoode]

Ah I see, I was thinking we would also be happy to make a PR to enable/disable the hashing comparison as a flag as well.

Does one settestmon_ignore_dependencies as a flag to the pytest job or within the pytest.ini, pyproject? You might very well be right in that the wheel built for the local library will always change its hash when we pack and install it somewhere else.

[jmkolbe]

@jacksongoode in case you're still wondering, the setting is accepted in pytest.ini as testmon_ignore_dependencies = '<package name>'

It would not accept a list on my end. Not sure if that's by design, it would seem reasonable to want to pass a list there. Also, the message

The packages installed in your Python environment have been changed. All tests have to be re-executed

is still shown.

But it does appear to fulfill the requirement, i.e. exempting specific installed packages when evaluating if the test run needs to be repeated 👍

[marcodicro-dp]

I'd also like this to be less sensitive.

E.g. If I built the testmon db this morning doing a full test run, then a version change in an external library like e.g google-cloud-bigquery doesn't mean I have to run all the tests again if I made a simple code change.

I've tried adding that library to testmon_ignore_dependencies in pytest.ini without success: testmon_ignore_dependencies = 'google-cloud-bigquery'

For this to work, does testmon_ignore_dependencies have to be set when the testmon DB is built too?