search

iiasa / ixmp

The ix modeling platform for integrated and cross-cutting scenario analysis
https://docs.messageix.org/ixmp
Apache License 2.0
38 stars 111 forks source link

Make pytest use one process for CI #510

Closed glatterf42 closed 8 months ago

glatterf42 commented 8 months ago

Using pytest-xdist is most likely causing flakiness in our test suite because some tests seem to write/read from processes they should not care about, leading to the common error messages listed below. While we ideally want to implement a more sustainable fix soon (e.g., making the names of the scenarios/processes unique), we don't want to spend a lot of time rerunning flaky tests manually in the meantime. So this PR makes pytest use just one process as a mitigation suggested by @khaeru.

Common error messages ```bash FAILED ixmp/tests/test_tutorials.py::test_py_transport - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ # launch the ix modeling platform using the default back end mp = ixmp.Platform() # The following lines have the same effect: # mp = ixmp.Platform(name='local') # Default local database # mp = ixmp.Platform(name='default') # Default database -> 'local' ------------------ --------------------------------------------------------------------------- Exception Traceback (most recent call last) File Platform.java:110, in at.ac.iiasa.ixmp.Platform.() File Platform.java:163, in at.ac.iiasa.ixmp.Platform.initPlatform() File DbDAO.java:238, in at.ac.iiasa.ixmp.database.DbDAO.create() File DbDAO.java:318, in at.ac.iiasa.ixmp.database.DbDAO.updateSchemaIfNecessary() File Flyway.java:910, in org.flywaydb.core.Flyway.migrate() File Flyway.java:1238, in org.flywaydb.core.Flyway.execute() File Flyway.java:910, in org.flywaydb.core.Flyway$1.execute() File Flyway.java:918, in org.flywaydb.core.Flyway$1.execute() File Flyway.java:78, in org.flywaydb.core.Flyway.access$100() File Flyway.java:1037, in org.flywaydb.core.Flyway.doValidate() Exception: Java Exception The above exception was the direct cause of the following exception: org.flywaydb.core.api.FlywayException Traceback (most recent call last) File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:297, in JDBCBackend.__init__(self, jvmargs, **kwargs) 296 try: --> 297 self.jobj = java.Platform("Python", properties) 298 except java.NoClassDefFoundError as e: # pragma: no cover org.flywaydb.core.api.FlywayException: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1 (hsql base version) During handling of the above exception, another exception occurred: RuntimeError Traceback (most recent call last) Cell In[2], line 2 1 # launch the ix modeling platform using the default back end ----> 2 mp = ixmp.Platform() 4 # The following lines have the same effect: 5 # mp = ixmp.Platform(name='local') # Default local database 6 # mp = ixmp.Platform(name='default') # Default database -> 'local' File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/core/platform.py:94, in Platform.__init__(self, name, backend, **backend_args) 88 raise ValueError( 89 f"backend class {repr(backend_class_name)} not among " 90 + str(sorted(BACKENDS.keys())) 91 ) 93 # Instantiate the backend ---> 94 self._backend = backend_class(**kwargs) File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:320, in JDBCBackend.__init__(self, jvmargs, **kwargs) 318 else: 319 _raise_jexception(e) --> 320 raise RuntimeError(f"{msg}\n(Java: {jclass})") 322 # Set the log level 323 self.set_log_level(log_level) RuntimeError: when initializing database: (Java: org.flywaydb.core.api.FlywayException) On: macos-latest-py3.8, macos-latest-py3.10 FAILED ixmp/tests/test_tutorials.py::test_py_transport_scenario - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ # launch the ix modeling platform using the default local database mp = ixmp.Platform() ------------------ --------------------------------------------------------------------------- Exception Traceback (most recent call last) File Platform.java:110, in at.ac.iiasa.ixmp.Platform.() File Platform.java:163, in at.ac.iiasa.ixmp.Platform.initPlatform() File DbDAO.java:238, in at.ac.iiasa.ixmp.database.DbDAO.create() File DbDAO.java:318, in at.ac.iiasa.ixmp.database.DbDAO.updateSchemaIfNecessary() File Flyway.java:910, in org.flywaydb.core.Flyway.migrate() File Flyway.java:1238, in org.flywaydb.core.Flyway.execute() File Flyway.java:910, in org.flywaydb.core.Flyway$1.execute() File Flyway.java:918, in org.flywaydb.core.Flyway$1.execute() File Flyway.java:78, in org.flywaydb.core.Flyway.access$100() File Flyway.java:1037, in org.flywaydb.core.Flyway.doValidate() Exception: Java Exception The above exception was the direct cause of the following exception: org.flywaydb.core.api.FlywayException Traceback (most recent call last) File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:297, in JDBCBackend.__init__(self, jvmargs, **kwargs) 296 try: --> 297 self.jobj = java.Platform("Python", properties) 298 except java.NoClassDefFoundError as e: # pragma: no cover org.flywaydb.core.api.FlywayException: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1 (hsql base version) During handling of the above exception, another exception occurred: RuntimeError Traceback (most recent call last) Cell In[2], line 2 1 # launch the ix modeling platform using the default local database ----> 2 mp = ixmp.Platform() File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/core/platform.py:94, in Platform.__init__(self, name, backend, **backend_args) 88 raise ValueError( 89 f"backend class {repr(backend_class_name)} not among " 90 + str(sorted(BACKENDS.keys())) 91 ) 93 # Instantiate the backend ---> 94 self._backend = backend_class(**kwargs) File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:320, in JDBCBackend.__init__(self, jvmargs, **kwargs) 318 else: 319 _raise_jexception(e) --> 320 raise RuntimeError(f"{msg}\n(Java: {jclass})") 322 # Set the log level 323 self.set_log_level(log_level) RuntimeError: when initializing database: (Java: org.flywaydb.core.api.FlywayException) On: macos-latest-py3.8, macos-latest-py3.10 FAILED ixmp/tests/test_tutorials.py::test_R_transport - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ # launch the ix modeling platform using a local HSQL database instance mp <- ixmp$Platform() ------------------ when initializing database: (Java: org.flywaydb.core.api.FlywayException)Traceback: 1. ixmp$Platform() 2. py_call_impl(callable, call_args$unnamed, call_args$named) On: macos-latest-py3.8, macos-latest-py3.10 FAILED ixmp/tests/test_tutorials.py::test_R_transport_scenario - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ # launch the ix modeling platform using the local default database mp <- ixmp$Platform() ------------------ when initializing database: (Java: org.flywaydb.core.api.FlywayException)Traceback: 1. ixmp$Platform() 2. py_call_impl(callable, call_args$unnamed, call_args$named) On: macos-latest-py3.8, macos-latest-py3.10 FAILED ixmp/tests/backend/test_jdbc.py::test_del_ts - assert 8 == (1 + 8) + where 8 = len() + where = .jindex + where = ._backend On: macos-latest-py3.12 FAILED ixmp/tests/core/test_timeseries.py::TestTimeSeries::test_last_update[TimeSeries] - assert datetime.timedelta(seconds=1, microseconds=390684) < datetime.timedelta(seconds=1) + where datetime.timedelta(seconds=1, microseconds=390684) = abs((datetime.datetime(2024, 1, 12, 6, 36, 10, 712000) - datetime.datetime(2024, 1, 12, 6, 36, 12, 102684))) + where datetime.datetime(2024, 1, 12, 6, 36, 12, 102684) = () + where = datetime.now + and datetime.timedelta(seconds=1) = timedelta(seconds=1) On: macos-latest-py3.12 FAILED ixmp/tests/core/test_platform.py::test_export_timeseries_data - AssertionError: DataFrame are different DataFrame shape mismatch [left]: (2, 10) [right]: (0, 10) On: macos-latest-py3.8, macos-latest-py3.12 FAILED ixmp/tests/backend/test_jdbc.py::test_connect_message - assert "connected to database 'jdbc:hsqldb:mem://ixmptest' (user: ixmp)..." in '' + where '' = CaptureResult(out='', err='').out On: windows-latest-py3.9, windows-latest-py3.12 FAILED ixmp/tests/backend/test_jdbc.py::test_del_ts - assert 8 == (1 + 8) + where 8 = len() + where = .jindex + where = ._backend On: macos-latest-py3.12 FAILED ixmp/tests/backend/test_jdbc.py::test_close - AssertionError: assert 'Database connection could not be closed or was already closed' in '' + where '' = CaptureResult(out='', err='').out On: windows-latest-py3.8 ```

How to review

PR checklist

codecov[bot] commented 8 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (e764f63) 98.9% compared to head (46aa2dc) 98.9%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #510 +/- ## ===================================== Coverage 98.9% 98.9% ===================================== Files 44 44 Lines 4794 4794 ===================================== Hits 4743 4743 Misses 51 51 ```