Unfavorable return value 0 in case of test result is UNKNOWN

[test-fullautomation]

In case of e.g. a version conflict the test result of the RobotFramework AIO is UNKNOWN. The return value is 0. This may cause misunderstandings because usually the value 0 indicates success.

It would be better to have an individual return value for tests with result UNKNOWN.

If Robot Framework Core doesn't already use the return value "2", then in case of "unknown" the execution should return 2.

[milanac030988]

In case of e.g. a version conflict the test result of the RobotFramework AIO is UNKNOWN. The return value is 0. This may cause misunderstandings because usually the value 0 indicates success.

It would be better to have an individual return value for tests with result UNKNOWN.

If Robot Framework Core doesn't already use the return value "2", then in case of "unknown" the execution should return 2.

Hello @test-fullautomation

• Current status: Robotframework returns a return code of min(self.suite.statistics.failed, 250), meaning the return code will be the minimum of the number of failed testcases and 250, and return 0 in case there is no test fail.

• Your expectation of return 2 for an unknown result may cause confusion with the case of 2 failed testcases.

• Proposed options:

  • Option 1: the return code will be the sum of failed and unknown tests. The issue is that users will not be able to determine how many tests fail and how many tests are unknown.

  • Option 2: the return code will be determined by the following code snippet: unknown_test = min(self.suite.statistics.unknown, 250) failed_test = min(self.suite.statistics.failed, 250) ret_val = (unknown_test << 8) | failed_test

By this way, users will be able to determine the number of failed and unknown tests by extracting the values from the combined integer through bit shifting.

Please let me know your opinion.

[HolQue]

Good analysis!

I would prefer the second option. But bit shifting is also rather tricky and causes additional effort ;-)

Question only: Might it be possible to use negative return values to indicate "UNKNOWN"?

In Test Trigger I do the same to distinguish between errors of system under test and test internal errors:

https://github.com/test-fullautomation/RobotFramework_AIO/blob/develop/test/documentation/ReturnValueHandling.xlsx

[milanac030988]

Might it be possible to use negative return values to indicate "UNKNOWN"?

Hello Holger,

That's a good point, Holger. But in the case of having both fails and unknowns, I haven't figured out how it will return a value for users to determine both with just one returned value.

[HolQue]

Yes, you are right. This option I overlooked. OK, then let's bit shifting ;-)

[test-fullautomation]

Hi Cuong,

Hm... another deviation from the OSS Version... I also agree to option 2. This is an elegant way to put both nr. failed and nr. unknown into the return value... and we know if not equal to 0 that something went wrong.

Thank you, Thomsa

[test-fullautomation]

solved and merged with https://github.com/test-fullautomation/robotframework/pull/19