search

iiasa / message-ix-models

Tools for the MESSAGEix-GLOBIOM family of models
https://docs.messageix.org/models
Apache License 2.0
16 stars 31 forks source link

Handling flaky tests #112

Closed glatterf42 closed 11 months ago

glatterf42 commented 11 months ago

For roughly the past month, I have collected data on flaky CI tests. The initial idea was to mark them as flaky, but as per pytest's docs on flaky tests, that should never be a long-term solution. Instead, tests should be (randomly) re-ordered, re-written for more atomic assertions, or split up into different groups to find the root cause of the flaky behavior and eliminate it. We will have to see when time permits this. For now, we could mark them as flaky to save us from re-running them manually. Here are the flaky tests of this repository I gathered so far:

Flaky tests

Auxiliary

Runtime error DB connection {#runtime-error-db-connection}

RuntimeError: unhandled Java exception: 
Unable to obtain connection from database (jdbc:hsqldb:file:/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/localdb/default) for user 'ixmp': Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@e3adcd5d[file =/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/localdb/default.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2023-07-24 05:25:35 heartbeat - read: -2209 ms.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State  : S1000
Error Code : -451
Message    : Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@e3adcd5d[file =/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/localdb/default.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2023-07-24 05:25:35 heartbeat - read: -2209 ms.

Names are shortened to message_ix_models/tests as the starting directory.

Test name Error message Runners (# of occurrences if > 1)
util/test_context.py::TestContext:: test_set_scenario Runtime error DB connection macos-latest-py3.11-upstream-v3.7.0, macos-latest-py3.11-upstream-main