iiasa / ixmp

The ix modeling platform for integrated and cross-cutting scenario analysis
https://docs.messageix.org/ixmp
Apache License 2.0
38 stars 111 forks source link

Handling flaky tests #490

Closed glatterf42 closed 1 year ago

glatterf42 commented 1 year ago

Closes #489 by marking tests as flaky. Through discussion with @khaeru, we agreed that this is the best option for now; should the tests make more issues in the future, we can still identify their root cause.

iiasa/message-ix-models#113 is waiting for this PR to be merged so that a temporary commit can be dropped there.

How to review

PR checklist

codecov[bot] commented 1 year ago

Codecov Report

Merging #490 (97f17da) into main (62a2a99) will increase coverage by 0.0%. The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #490   +/-   ##
=====================================
  Coverage   98.5%   98.5%           
=====================================
  Files         42      42           
  Lines       4489    4506   +17     
=====================================
+ Hits        4425    4442   +17     
  Misses        64      64           
Files Changed Coverage Δ
ixmp/backend/jdbc.py 95.2% <100.0%> (ø)
ixmp/tests/backend/test_jdbc.py 100.0% <100.0%> (ø)
ixmp/tests/test_access.py 100.0% <100.0%> (ø)
ixmp/tests/test_integration.py 100.0% <100.0%> (ø)
ixmp/tests/test_tutorials.py 100.0% <100.0%> (ø)
glatterf42 commented 1 year ago

The first attempt at windows-latest-py3.8 failed exactly with one of the errors that I hoped to avoid by marking tests as flaky and I can't find any sign in the logs that would indicate multiple reruns, which seems odd to me. Since GITHUB_ACTIONS should definitely be set in os.environ, I'm wondering whether sys.platform == "win32" is too restrictive. The recommended way to test for the platform might be different: import platform; platform.system() prints 'Windows', 'Linux', or 'Darwin'. I will try this tomorrow.