Retry only failed test when --num-flaky-test-attempts is provided

[jerrycp]

GIVEN --num-flaky-test-attempts is N, WHEN flank is run , THEN

• expect that only failing test is retried up to N times.
• expect that passing test is executed once.

I have tried running flank v23.07.0, and provided the argument --num-flaky-test-attempts=4, for 2 test named as :

• alwaysPass
• alwaysFail

I expected the "alwaysPass" test to be executed only once, without further retry. But, it was executed for 4 times as shown in the attached screenshot.

Based on my understanding of the previous issue : https://github.com/Flank/flank/issues/636 I believe that the retry feature should only be executed on failing test ?

Can someone help to provide advice ?

[adamvduke]

The linked issue points to Flank delegating the retry logic to the Firebase backend, but the documentation doesn't concretely describe exactly what will be re-run, e.g. entire execution, shard, individual test case. I opened a conversation internally with folks to investigate.

[adamvduke]

Following up, the behavior is working as intended. The documentation specifies that the execution will be retried. In certain cases that might mean retrying a shard instead of the entire set of tests, but explicitly does not mean retrying individual test cases. I'll work to get the documentation clarified.