Add test failure severity support

[vnadot]

It could be nice to have severity on test failure.

My first thought is that, like the EPICS alarms, we could have "minor" and "major" alarms. The default alarm would be "major" if the severity keyword not is provided.

- name:       "my test with a low severity"
      getter:     "${P}myPv"
      get_value:  1000  # ns
      delta:      200   # ns
      skip:       "${skip_test0_pass}"
      severity: minor

I would say that by default, if a minor alarm is raised and "on_failure" is set to "stop" then test execution should keep going. However, we should have a way to also stop on minor alarm.

To be discussed !

[gohierf]

Hi, the current behavior is something along the lines:

on_failure: determines whether to continue, pause or abort if the test fails

And the default settings can be set for the whole file.

Isn't abort the equivalent of major severity and continue the equivalent of "minor" ? Plus there is the intermediate option to pause for user decision on whether to continue or abort.

I feel like adding serverity is:

• already covered by on_failure
• subject to confusion with the tradionnal meaning of severity of EPICS alarms.

So I'm rather against this request.

[gohierf]

In your example:

- name:       "my test with a low severity"
      getter:     "${P}myPv"
      get_value:  1000  # ns
      delta:      200   # ns
      skip:       "${skip_test0_pass}"
      severity: minor

You could have instead:

- name:       "my test with a low severity"
      getter:     "${P}myPv"
      get_value:  1000  # ns
      delta:      200   # ns
      skip:       "${skip_test0_pass}"
      on_failure: "${minor}"

And in the variables define minor as continue, pause or abort, as you wish for this file.

[vnadot]

I have some cases where I want to continue in any error but I would like to easely monitor the minor alarms with the major alarms for better readability of the report test.