Tracking flaky tests

[czeslavo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Problem Statement

Currently, we're missing data points to tell exactly which test cases tend to fail occasionally while running them against the main branch (being "flaky"). It makes tracking the individual flaky test cases harder as it requires manual issue creation while somebody spots a failure. It also makes prioritizing harder due to not being able to tell how often the test case fails (given it fails very rarely, we could assign it a lower priority).

As a project maintainer, I'd like to have insights on the healthiness of test suites running against the main branch.

Proposed Solution

• track test suites history by exporting JUnit xmls and storing them as build artifacts
• find a tool that will help with visualizing the test suites' healthiness (examples: https://github.com/F-Secure/flaky-tests-detection, https://github.com/marketplace/actions/buildpulse)
• integrate the chosen tool with our workflows to provide basic visibility
• (optional) automatically create a GH issue tracking a flaky test once it is detected
• (optional) link to every pipeline failure due to the detected flaky test case in the GH issue

Additional information

Paper on the topic that's probably worth looking into: "Modeling and ranking flaky tests at Apple".

Acceptance Criteria

• [x] KIC's workflows (test.yaml, nightly.yaml, e2e.yaml) are configured to report test results to BuildPulse on every run against main branch
• [x] BuildPulse displays summaries regarding particular tests' flakiness for KIC

[pmalek]

https://buildpulse.io/ 😲

Stop losing time to flaky tests

Where have you been for so long buildpulse.

[czeslavo]

Yeah, the buildpulse seems to be the easiest solution, that's the one I mentioned during retro - it's paid $99 a month and has 14d free trial.

[czeslavo]

We've decided to go with BuildPulse trial, I've updated the issue's ACs to reflect that.

[czeslavo]

Closing as it's been implemented in KIC.