reframe-hpc / reframe

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.
https://reframe-hpc.readthedocs.org
BSD 3-Clause "New" or "Revised" License
213 stars 98 forks source link

Using `--restore-session` fails when some tests were skipped in the previous run #3219

Open NMannall opened 1 week ago

NMannall commented 1 week ago

When running with --restore-session and --failed, the following error occurs if any of the tests were skipped in the run being restored:

Traceback (most recent call last):
  File "/mnt/.../.venv/lib/python3.9/site-packages/reframe/frontend/runreport.py", line 159, in _load_report
    jsonschema.validate(report, schema)
  File "/mnt/.../.venv/lib/python3.9/site-packages/jsonschema/validators.py", line 1312, in validate
    raise error
jsonschema.exceptions.ValidationError: 'skipped' is not one of ['success', 'failure', 'aborted']

Failed validating 'enum' in schema['properties']['runs']['items']['properties']['testcases']['items']['properties']['result']:
    {'enum': ['success', 'failure', 'aborted'], 'type': 'string'}

On instance['runs'][0]['testcases'][0]['result']:
    'skipped'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/.../.venv/bin/reframe", line 23, in <module>
    cli.main()
  File "/mnt/.../.venv/lib/python3.9/site-packages/reframe/core/logging.py", line 1034, in _fn
    return fn(*args, **kwargs)
  File "/mnt/.../.venv/lib/python3.9/site-packages/reframe/frontend/cli.py", line 918, in main
    report = runreport.load_report(*filenames)
  File "/mnt/.../.venv/lib/python3.9/site-packages/reframe/frontend/runreport.py", line 176, in load_report
    rpt = _load_report(primary)
  File "/mnt/.../.venv/lib/python3.9/site-packages/reframe/frontend/runreport.py", line 166, in _load_report
    raise errors.ReframeError(
reframe.core.exceptions.ReframeError: invalid report '/home/.../.reframe/reports/run-report-296.json' (required data version: 3.1), found: 3.1): 'skipped' is not one of ['success', 'failure', 'aborted']

Failed validating 'enum' in schema['properties']['runs']['items']['properties']['testcases']['items']['properties']['result']:
    {'enum': ['success', 'failure', 'aborted'], 'type': 'string'}

On instance['runs'][0]['testcases'][0]['result']:
    'skipped'
vkarak commented 1 week ago

Indeed, this is a bug. I could reproduce it. Steps to reproduce:

  1. Change the hook of SkipTest in unittests/resources/checks/hellocheck.py to post-setup
  2. Force a failure in HelloTest:
reframe -c unittests/resources/checks/hellocheck.py -S HelloTest.prebuild_cmds=false -r
  1. Try restoring the session
reframe -c unittests/resources/checks/hellocheck.py --restore-session --failed -r