search

operator-framework / operator-lifecycle-manager

A management framework for extending Kubernetes with Operators
https://olm.operatorframework.io
Apache License 2.0
1.69k stars 543 forks source link

Define a process for eliminating flakes #2590

Open perdasilva opened 2 years ago

perdasilva commented 2 years ago

Description

As a maintainer or contributor I don't want to have to keep dealing with flaky tests and I also don't want to introduce any new flaky tests. We should have a process whereby is e2e tests are added or changed we'd need to have a no-flake certification

Acceptance Criteria

  1. Anti-flake process defined and documented
  2. PR checklist updated
  3. Anti-flake process is push button (e.g. make flake-detection test1 test2 test3 --pr=1234)
fgiloux commented 2 years ago

Although it does not happen very often it may be worth considering in the process that a test may not have been borne but may become flaky due to changes made to the code base (introduction of a race condition or something blocking the deletion of a resource or the tear-down of the environment).