Automatic Test Filtering - avoid flaky tests

[ghost]

CI/CD/CT - Pipelines depend on running automated tests to verify the code build.

Challenge: Flaky automated tests report code build is bad when the automated test is actually bad and not the code.

Request: Execute KS Test Suite and automatically skip any test case that is flaky (Greater than X% flaky). Add a new field on the Test Suite Execution screen that sets the flakiness limit. Similar to the retry count, the flakiness limit would apply during execution.

Benefits: Executes only the best test cases to verify code build. If test flakiness goes up or down then additional tests are executed or skipped - automatically. Everyone can rely on the test automation because flaky tests are never executed. That improves trust and faith that the test automation adds value. If a test fails due to a bug, then the chances are the bug is real because the test was not flaky.

How

1. Add flakiness limit on test execution screen.
2. Create an API call that returns a test's "flakiness".
3. If test "flakiness" > 25% then "skip" test. (25% is just an example)
4. Repeat for each test.

This could also be done in the BeforeTestListener class if the API call were made public. In that case, no need to add to the Test Execution. However, test execution screen would be easier for all users.

[nhmy]

Hello @jimsears. We will consider your suggestion and add it to our backlog. However, this feature requires further assessment and planning, hence, will not be available this year. We will keep you updated when it's available. Thank you!