wrong state "unknown" in case of fatal error

[test-fullautomation]

If a fatal error occurs during a test execution, the current test case is set to 'Unknown' and the test execution is terminated. Any test cases that were supposed to be executed subsequently will be skipped and will have the status 'Unknown' (even though they were not executed). Intuitively, we would expect the test cases to be set to 'Skipped'.

The return value shall reflect the number of "skipped" test cases, too.

[HolQue]

I gave it a try.

A robot file contains 4 test cases. The second test case contains the keyword FATAL ERROR. Outcome: Test case 1: PASSED Test case 2: FAILED Test case 3: UNKNOWN Test case 4: UNKNOWN

This is announced in console: 4 tests, 1 passed, 1 failed, 2 unknown And in XML log file: <stat pass="1" fail="1" skip="0" unknown="2">All Tests</stat>

Is my understanding right, that in opposite to the above this is the desired alternative behavior:

Test case 1: PASSED Test case 2: FAILED Test case 3: SKIPPED Test case 4: SKIPPED

This is announced in console: 4 tests, 1 passed, 1 failed, 2 skipped And in XML log file: <stat pass="1" fail="1" skip="2" unknown="0">All Tests</stat>

?

Or shall the result of the test case with the fatal error be UNKNOWN (instead of FAILED)?

[HolQue]

Up to now the return value of the Robot Framework considers FAILED and UNKNOWN:

ret_val = (unknown_test << 8) | failed_test

If we introduce SKIPPED: Shall the number of skipped test cases be added to the return value computation?

[HolQue]

See test case TSM_0559, available with: https://github.com/test-fullautomation/robotframework-testsuitesmanagement/pull/174

[test-fullautomation]

Hi Holger, @milanac030988,

I got this request from Denis and was also not clear what should be the result. But after some thinking now: "unknown" means that there was a bug in the test case, therefore unknown also implies that it was tried to execute the test, but the test didn't work. Finally it suggests that something is wrong with the test case. "skipped" would be much better in this case as the tests were really not executed at all. So we don't know anything about the test case and the feature which was tested with this test case.

=> Expectation is Test case 1: PASSED Test case 2: UNKNOWN Test case 3: SKIPPED Test case 4: SKIPPED

Thank you, Thomas

[HolQue]

Hi Thomas,

this is plausible for me. But I have one concern:

Test case 2 is the one with the fatal error. The test case has been executed, but only up to the position of the fatal error. "fatal" means, we have an issue with the test itself. Therefore this test case is an UNKNOWN candidate.

Accordingly to your definition of UNKNOWN in my opinion this is the expectation:

Test case 1: PASSED
Test case 2: UNKNOWN
Test case 3: SKIPPED
Test case 4: SKIPPED

[test-fullautomation]

Yes Holger, you are right. When I copied the sequence I overlooked the "Failed".

As you write:

Test case 1: PASSED
Test case 2: UNKNOWN
Test case 3: SKIPPED
Test case 4: SKIPPED

would be my Expectation.

I updated also my comment from before (3:06) to avoid confusion.