Changes > 1.4.5 made testmon more sensitive to differently packaged environments?

[jacksongoode]

We have a system where we send a wheel to install all of our version locked packages on a Spark cluster. It seems that since 1.4.5 (we tested on 2.0.6) there was some change that made testmon more sensitive and a new db is being created (without any other message than just new db) while the old testmondata file that is passed into the environment seems to be ignored. It's odd because this testmondata file was created from an instance of the same machine, though the Python packages do get setup every time.

Wondering if there are any insights you have about what might've changed between the versions?

[pokopt]

Hi Jackson, let me clarify. Did you try to use testmon data file from v1.4.5 together with testmon v2.0.6? If this is the case unfortunately there are some breaking changes in data format and new version can't use old data file and simply recreates the database. If I didn't get you right I'll kindly ask you for more details.

[jacksongoode]

Hi @pokopt,

No this was actually with fresh testmondata files from 2.0.6.

[jacksongoode]

@pokopt Any further insights? We just had to roll back to 1.4.5 given the new sensitivity.

[tarpas]

Jackson,

short answer to your question is no. At least not intentionally. There was a refactoring of the functionality, but the behaviour should've stayed the same. Do you use testmon environments? (via testmon-env or environment_expression)

[tarpas]

did the downgrade to 1.4.5 help?

[tarpas]

@jacksongoode There is beta version in PYPI now which disregards changes in dependencies version after the 2nd dot, meaning the data which has been created with e.g. dependency==1.0.0 will be reused for all runs with dependency==1.0.x ... please install with pip install --pre pytest-testmon and let me know any thoughts.

[jacksongoode]

@tarpas That sounds great, once that becomes stable we'll jump on it and I'll let you know. Reverting to 1.4.5 did resolve our issues.

[tarpas]

@jacksongoode It's stable now. But since you're getting new DB it will not solve your problem. if you run $ sqlite3 .testmondata and select distinct filename from file_fp; you should see files of your project with paths relative to rootdir of your pytest run. Do you?

[tarpas]

Closing this because there has been fixes in this area and there is no activity from OP.

[jacksongoode]

Sorry to reopen this but I tried to update from 1.4.5 to 2.1.0, created a new testmon file, ran tests, and after changing some code, testmon thinks its a "newDB" and reruns all tests.

[jacksongoode]

Just to be clear how we install libraries, we use a Pipfile.lock file (from pipenv) and install the dependencies in a remote environment with the versions of each package. We then select some tests and save the resultant .testmondata file. When we start to run our tests that .testmondata is taken from the last cache and restored. Maybe the architecture causes some change that testmon became sensitive to after 1.4.5.

Edit: Realizing that .*-shm and .*-wal are created as well now. Should these be cached and recovered for tests? I believe that might be it!

[tarpas]

Edit: Realizing that .*-shm and .*-wal are created as well now. Should these be cached and recovered for tests? I believe that might be it!

Yes, that's quite probable. I'll think of disabling WAL or doing PRAGMA wal_checkpoint(TRUNCATE);

I think only .*-wal file has information. You can try to copy it too or execute sqlitecommand PRAGMA schema.wal_checkpoint(TRUNCATE); so that .testmondata includes everything.

[jacksongoode]

Hmm, well I'm glad this discovery resolved the issue. It would be nice if there were fewer files to watch 👍