search

OpenMDAO / testflo

A simple python testing framework that can run unit tests under MPI (or not).
Other
3 stars 7 forks source link

Unit tests with the skip decorator fail in Python 3.12.1 #99

Closed crecine closed 11 months ago

crecine commented 11 months ago

A unit test using the skip decorator fails in Python 3.12.1, but not in previous Python versions

@unittest.skip('Skipping this benchmark')
def test_cruise_result(self):

The following is the result of running that test case from testflo in Python 3.12.1

test_cruise.py:TestCruise.test_cruise_result  ... FAIL (-6:-17:-19.87, 0 MB)
Traceback (most recent call last):
  File "/home/crecine/miniconda3/envs/dev_env/lib/python3.12/site-packages/testflo/test.py", line 64, in testcontext
    yield
  File "/home/crecine/miniconda3/envs/dev_env/lib/python3.12/site-packages/testflo/test.py", line 337, in run
    parent.run(result)
  File "/home/crecine/miniconda3/envs/dev_env/lib/python3.12/unittest/case.py", line 617, in run
    _addSkip(result, self, skip_why)
  File "/home/crecine/miniconda3/envs/dev_env/lib/python3.12/unittest/case.py", line 90, in _addSkip
    addSkip(test_case, reason)
  File "/home/crecine/miniconda3/envs/dev_env/lib/python3.12/site-packages/testflo/utresult.py", line 81, in addSkip
    resdata = self._tests[test.id()]
              ~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'aviary.mission.gasp_based.phases.benchmark.test_cruise.TestCruise.test_cruise_result'

In prior versions of Python, the test skipped as expected.

test_cruise.py:TestCruise.test_cruise_result  ... SKIP (00:00:0.00, 174 MB)
Skipping this benchmark

The following tests were skipped:
test_cruise.py:TestCruise.test_cruise_result

Running the test directly with unittest.main() also skips as expected.

Ran 1 test in 0.000s

OK (skipped=1)

In the recent change to unittest, result.startTest was moved from the beginning of the run method of TestCase to after the skip decorator is handled. image

UnitTestResult appears to rely on this being called to populate self._tests which is then used in its addSkip method.

crecine commented 11 months ago

using the update in the linked PR allows test to skip successfully.

test_cruise.py:TestCruise.test_cruise_result  ... SKIP (00:00:0.00, 161 MB)
Skipping this benchmark

The following tests were skipped:
test_cruise.py:TestCruise.test_cruise_result