search

iiasa / message_ix

The integrated assessment and energy systems model MESSAGEix
https://docs.messageix.org
Apache License 2.0
112 stars 149 forks source link

Flaky tests and pytest-xdist #776

Closed glatterf42 closed 4 months ago

glatterf42 commented 5 months ago

@glatterf42 FYI, looking at the recent commits on the branch, I am finding that for some apparently flaky tests, it helps to .clone(scenario=request.node.name) first.

I think this is because we are using pytest-xdist: where this results in 2+ tests with identical model name and scenario names being run in parallel, then strange effects can occur; for instance, one test reads and sees the result from the GDX file of the other test. (This occurs because message_ix tries to heed users' desire to keep GDX I/O files all in the same directory, rather than separate, temporary directories per-run as the base ixmp GAMSModel class does.)

request.node.name is from the Pytest request fixture, and gives a string that is unique to each test—including individual cases of parametrized tests. It's the easiest way to get such a unique label.

We could later (no urgency) try this approach to address other apparently flaky tests.

_Originally posted by @khaeru in https://github.com/iiasa/message_ix/issues/765#issuecomment-1851995857_

We might be able to resolve some FLAKY markers in our code by following this advice. Also, I'm collecting information again on which tests currently appear to be flaky.