search

epics-extensions / WeTest

Test automation utility for EPICS modules (from YAML configuration to PDF test reports)
Other
8 stars 2 forks source link

add a hint list to help user understand the wetest error #85

Open vnadot opened 7 months ago

vnadot commented 7 months ago

WeTest errors are not very user-friendly. For example (they are many examples):

for this test:

    - name:       "enable PV put (DISP EPICS field)"
      setter:     "SL-SCL-CM1:RF-RFFI-1:GenDriverScanSet.DISP"
      set_value:  "0"

I got this error message

ERROR: test-0-0-1 (wetest.testing.generator.SelectableTestCase)
enable PV put: enable PV put (DISP EPICS field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/testing/generator.py", line 438, in test
    setter.put(set_value)
  File "/iee/extensions/lib/python/pyepics-3.2.4-py2.7.egg/epics/pv.py", line 336, in put
    return ca.put(self.chid, value,
  File "/iee/extensions/lib/python/pyepics-3.2.4-py2.7.egg/epics/ca.py", line 403, in wrapper
    return fcn(*args, **kwds)
  File "/iee/extensions/lib/python/pyepics-3.2.4-py2.7.egg/epics/ca.py", line 1329, in put
    PySEVCHK('put', ret)
  File "/iee/extensions/lib/python/pyepics-3.2.4-py2.7.egg/epics/ca.py", line 432, in PySEVCHK
    raise CASeverityException(func_name, message(status))
epics.ca.CASeverityException:  put returned 'Invalid element count requested'

The reason is because of a type error (see #84). Setting set_value: 0 in my case fix the issue.

I propose to add some hints after an error message. In this specific case it could be: type mismatch: try to set a string in int type

vnadot commented 7 months ago

Another example of not user-friendly message: I run a suite and I don't know which file does not meet the version requirements.

$ ./${WETEST_PATH}/${WETEST_NAME} test/misc/WeTest/epicsComGen/rffi_WeTest_register_map_test_epicsComGen.yaml  --no-gui  -m P=SL-SCL-CM1:RF-RFFI-1: --propagate-macros
Traceback (most recent call last):
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/bin/..wetest-wrapped-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/command_line.py", line 426, in main
    suite, configs = generate_tests(
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/command_line.py", line 171, in generate_tests
    tests_data = ScenarioReader(
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/testing/reader.py", line 343, in __init__
    self.deserialized = self._deserialize()
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/testing/reader.py", line 437, in _deserialize
    new_sc = ScenarioReader(
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/testing/reader.py", line 350, in __init__
    self.version_is_supported = ScenarioReader.supports_version(
  File "/nix/store/pnkbr2dx35h9rz9zn3fgvcgvfr73smvs-python3.9-wetest-2.0.0rc1/lib/python3.9/site-packages/wetest/testing/reader.py", line 723, in supports_version
    raise UnsupportedFileFormatError(file_version)
wetest.testing.reader.UnsupportedFileFormatError: Scenario version '1.2.0' not supported. Current WeTest version: '2.0.0'.
Look at the WeTest repository for the CHANGELOG:
https://github.com/epics-extensions/WeTest

I would like a message like this:

wetest.testing.reader.UnsupportedFileFormatError: Scenario version '1.2.0' (file toto.yaml) not supported.