codecov-commenter
commented
2 months ago Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 98.87%. Comparing base (
f00caf6) to head (5e2a950).
Additional details and impacted files
```diff @@ Coverage Diff @@ ## main #212 +/- ## ========================================== + Coverage 98.84% 98.87% +0.02% ========================================== Files 253 256 +3 Lines 11630 11868 +238 Branches 521 533 +12 ========================================== + Hits 11496 11734 +238 Misses 134 134 ```:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
What does this PR do? This PR adds auto test retries support for cucumber test framework. The feature is enabled if
flaky_test_retries_enabledfield is set in remote settings.As Cucumber has its own built-in retries mechanism, it is enough for us to override the framework configuration and enable test retries when this feature is enabled on Datadog side.
The following customer-facing settings are added:
DD_CIVISIBILITY_RETRY_FAILED_TESTS_MAX_ATTEMPTS- number of test retries per each failed test (default: 5)DD_CIVISIBILITY_RETRY_FAILED_TESTS_TOTAL_LIMIT- total number of failed tests that are going to be retried (default: 100)Also this PR fixes retries support for test suite and test sessions status: now test suite and test session are reported as passed if tests succeed on retries.
How to test the change? Unit tests are provided for all the possible cases when retrying failed tests.