The Sauce platform distinguishes failures from errors. Failures to tend indicate test failures, aka assertion failures. Errors on the other hand are more “fatal” and pertain to the execution of the Job itself.
Our investigation has shown that if a job errors, it's safer to not rely on a test framework's test report for the specific use case of failed-test-filtering via smart retry, because the report itself may not contain all tests that were supposed to run and only results up to the point of failure. Saucectl would then retry the failed test(s), but the unexecuted tests will remain unexecuted.
With this change, smart retry will retry errored jobs much a like a regular retry, without any "smart filtering".
Description
The Sauce platform distinguishes failures from errors. Failures to tend indicate test failures, aka assertion failures. Errors on the other hand are more “fatal” and pertain to the execution of the Job itself.
Our investigation has shown that if a job errors, it's safer to not rely on a test framework's test report for the specific use case of failed-test-filtering via smart retry, because the report itself may not contain all tests that were supposed to run and only results up to the point of failure. Saucectl would then retry the failed test(s), but the unexecuted tests will remain unexecuted.
With this change, smart retry will retry errored jobs much a like a regular retry, without any "smart filtering".