iiasa / message_ix

The integrated assessment and energy systems model MESSAGEix
https://docs.messageix.org
Apache License 2.0
111 stars 149 forks source link

Make more tests robust & independent #797

Open glatterf42 opened 3 months ago

glatterf42 commented 3 months ago

Unfortunately, even though #784 improved the tests a lot, we still seem to see flaky behaviour. So let's continue the same kind of work from #784 on these tests:

glatterf42 commented 3 months ago

It seems like this flaky behaviour wasn't completely resolved by the first PR:

glatterf42 commented 3 months ago

In addition, I'm seeing another error for a test that we thought we made non-flaky:

This is happening for macos-latest-py3.8, macos-latest-py3.9, and macos-latest-py3.10, but for 3.9 and 3.10, the test case in question is rather [299-2] (and everything else is exactly the same). Edit: now this is also happening with [299-3] on macos-latest-py3.9 and [299-1] on macos-latest-py3.10.

glatterf42 commented 3 months ago

There's a new flaky error for test_add_year:

glatterf42 commented 3 months ago

Another one bites the dust:

glatterf42 commented 1 month ago

Another flaky test appeared today:

glatterf42 commented 1 month ago

There goes the next one:

glatterf42 commented 1 month ago

Postponing as internal PRs are not that time-sensitive anyway.

glatterf42 commented 1 month ago