search

pl-strflt / ipdx

2 stars 0 forks source link

Automate disabling flaky tests #60

Open galargh opened 1 year ago

galargh commented 1 year ago

Flaky test lowers make CI unpredictable. You have to review the CI run and compare it with other runs to actually accept a PR (for example with bitswap PR).

https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/

Things we can do now:

Monitor CI run and identify flaky tests for go-bitswap (the one case I, Laurent, knows about)

Monitor CI runs and identify flaky tests for project xxxx

Things that would help:

Reach out to maintainers and raise the discussion about disabling flaky tests entirely

Things we can do later

Automate Flakyness measure & report

Related PRs

https://github.com/ipfs/go-libipfs/issues/86

galargh commented 1 year ago

This should build up and be linked to Marco's work on flaky tests in go-libp2p.