Open glatterf42 opened 3 months ago
It seems like this flaky behaviour wasn't completely resolved by the first PR:
ERROR message_ix/tests/tools/test_add_year.py::test_add_year - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (BOI) not found in GDX file
On: windows-latest-py3.8
In addition, I'm seeing another error for a test that we thought we made non-flaky:
ERROR message_ix/tests/test_feature_bound_activity_shares.py::test_b_a_u_all_modes[299-5] - FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/message_ix/message_ix/message_ix/model/225a'
On: macos-latest-py3.8, macos-latest-py3.9, macos-latest-py3.10, ubuntu-latest-py3.9, ubuntu-latest-py3.11
This is happening for macos-latest-py3.8, macos-latest-py3.9, and macos-latest-py3.10, but for 3.9 and 3.10, the test case in question is rather [299-2]
(and everything else is exactly the same).
Edit: now this is also happening with [299-3]
on macos-latest-py3.9 and [299-1]
on macos-latest-py3.10.
There's a new flaky error for test_add_year
:
ERROR message_ix/tests/tools/test_add_year.py::test_add_year_cli - RuntimeError: unhandled Java exception: Error opening GDX file: Expected data marker (SYMBOL_2) not found in GDX file
On: windows-latest-py3.10
Another one bites the dust:
FAILED message_ix/tests/test_cli.py::test_copy_model - AssertionError: assert 1 == 0
+ where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
On: macos-latest-py3.9, macos-latest-py3.11
Another flaky test appeared today:
FAILED message_ix/tests/test_util.py::test_testing_make_scenario - RuntimeError: unhandled Java exception: The index set 'emission' does not have an element 'CO2'!
On: macos-13-py3.10
There goes the next one:
FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_add_share_mode_lo - assert False
+ where False = <function isclose at 0x000001BF5AF800F0>(0.5416666666666666, 0.525)
+ where <function isclose at 0x000001BF5AF800F0> = np.isclose
On: windows-latest-py3.11
Postponing as internal PRs are not that time-sensitive anyway.
FAILED message_ix/tests/test_feature_bound_activity_shares.py::test_commodity_share_lo - assert False
+ where False = <function isclose at 0x10fe1d8b0>(0.0, 0.15384615384615385)
+ where <function isclose at 0x10fe1d8b0> = np.isclose
+ and 0.15384615384615385 = <function test_commodity_share_lo.<locals>.calc_share at 0x161a71ca0>(<message_ix.core.Scenario object at 0x158b22370>)
On: macos-latest-py3.8
Unfortunately, even though #784 improved the tests a lot, we still seem to see flaky behaviour. So let's continue the same kind of work from #784 on these tests: