EESSI / compatibility-layer

Compatibility layer of the EESSI project
https://eessi.github.io/docs/compatibility_layer
GNU General Public License v2.0
12 stars 21 forks source link

ReFrame tests for compat layer are broken due to using test parameters in class body #145

Closed boegel closed 2 years ago

boegel commented 2 years ago

The worst part is that ReFrame just ignores the "syntax error", and then concludes with success (because all zero tests have passed)...

eessi@ip-172-31-18-112 ~/compatibility-layer $ reframe -r -v -c ./test/compat_layer.py
[ReFrame Setup]
  version:           3.9.1
  command:           '/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python-exec/python3.9/reframe -r -v -c ./test/compat_layer.py'
  launched by:       ec2-user@ip-172-31-18-112.eu-west-1.compute.internal
  working directory: '/home/ec2-user/compatibility-layer'
  settings file:     '<builtin>'
  check search path: '/home/ec2-user/compatibility-layer/test/compat_layer.py'
  stage directory:   '/home/ec2-user/compatibility-layer/stage'
  output directory:  '/home/ec2-user/compatibility-layer/output'

/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python-exec/python3.9/reframe: skipping test file '/home/ec2-user/compatibility-layer/test/compat_layer.py': reframe syntax error: accessing a test parameter from the class body is disallowed (rerun with '-v' for more information)
Traceback (most recent call last):
  File "/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python3.9/site-packages/reframe/frontend/loader.py", line 237, in load_from_file
    util.import_module_from_file(filename, force)
  File "/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python3.9/site-packages/reframe/utility/__init__.py", line 91, in import_module_from_file
    return _do_import_module_from_file(filename, module_name)
  File "/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python3.9/site-packages/reframe/utility/__init__.py", line 68, in _do_import_module_from_file
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/ec2-user/compatibility-layer/test/compat_layer.py", line 11, in <module>
    class RunInGentooPrefixTest(rfm.RunOnlyRegressionTest):
  File "/home/ec2-user/compatibility-layer/test/compat_layer.py", line 17, in RunInGentooPrefixTest
    if eessi_version != '2021.06':
  File "/cvmfs/pilot.eessi-hpc.org/2021.12/compat/linux/x86_64/usr/lib/python3.9/site-packages/reframe/core/meta.py", line 143, in __getitem__
    raise ReframeSyntaxError(
reframe.core.exceptions.ReframeSyntaxError: accessing a test parameter from the class body is disallowed

Loaded 0 test(s)
Generated 0 test case(s)
Filtering test cases(s) by name: 0 remaining
Filtering test cases(s) by tags: 0 remaining
Filtering test cases(s) by other attributes: 0 remaining
Final number of test cases: 0
[==========] Running 0 check(s)
[==========] Started on Tue Nov 30 07:52:29 2021

[----------] waiting for spawned checks to finish
[----------] all spawned checks have finished

[  PASSED  ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 skipped)
[==========] Finished on Tue Nov 30 07:52:29 2021
Run report saved in '/home/ec2-user/.reframe/reports/run-report.json'
Log file(s) saved in '/tmp/rfm-m33a1fsh.log'
boegel commented 2 years ago

Upstream issue regarding exit code 0 even when there are syntax errors: https://github.com/eth-cscs/reframe/issues/2302