Closed anmarchenko closed 2 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 98.87%. Comparing base (
f00caf6
) to head (5e2a950
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
What does this PR do? This PR adds auto test retries support for cucumber test framework. The feature is enabled if
flaky_test_retries_enabled
field is set in remote settings.As Cucumber has its own built-in retries mechanism, it is enough for us to override the framework configuration and enable test retries when this feature is enabled on Datadog side.
The following customer-facing settings are added:
DD_CIVISIBILITY_RETRY_FAILED_TESTS_MAX_ATTEMPTS
- number of test retries per each failed test (default: 5)DD_CIVISIBILITY_RETRY_FAILED_TESTS_TOTAL_LIMIT
- total number of failed tests that are going to be retried (default: 100)Also this PR fixes retries support for test suite and test sessions status: now test suite and test session are reported as passed if tests succeed on retries.
How to test the change? Unit tests are provided for all the possible cases when retrying failed tests.