A standard action for detecting flaky tests

[andy-sweet]

Description

This is a request to add a standard GitHub action for detecting and reporting flaky tests. By flaky test, I mean one that only sometimes passes, but sometimes fails, even when using the exact same commit/environment. These are often caused by tests that execute multiple threads/processes, but wait for fixed/hopeful periods of time instead of waiting for tasks to finish. For example, see a recent test failure on napari:main which is unlikely to be caused by the merged PR, and passed in the previous run on main.

Motivation

These are frustrating for developers because they block PRs from getting merged, make it harder to understand what failures your changes caused, and in extreme cases can lead to a lack of trust in test results. Overall, they tend to slow down development.

By detecting these flaky tests, we can do several useful things.

1. Understand how often they fail.
2. Temporarily skip some of the more flaky tests to unblock other developers.
3. Focus on fixing these tests to not be flaky (e.g. by waiting for non-main threads to finish).

Ideas

After a very brief search on the GitHub actions market place, I found a few possible candidates. Some of these are tied to services like BuildPulse and Testspace. A more promising candidate for a community driven effort like napari could be something like Shaker.

[jakeyheath]

@edulop91 and @alexlokshin-czi