search

sky-uk / osprey

Kubernetes OIDC CLI login
BSD 3-Clause "New" or "Revised" License
50 stars 18 forks source link

e2e tests are flaky when run in parallel #66

Closed aecay closed 3 years ago

aecay commented 3 years ago

When running the e2e tests locally, I observed failures which were flaky (in the sense that a different test/step would fail each time) but fairly consistent (in the sense that any given test run was pretty likely to fail). Explicitly disabling parallelism with go test -p 1 removed these failures. I suspect tests in different modules are fighting over ports (or some other global resource). If I'm correct in my hypothesis that port numbers are the contended resource, then probably global port allocation needs to be used across the whole test suite like this.

(NB this is not just ports for the go code to bind to, but also potentially ports for the LDAP server we spin up...)

aecay commented 3 years ago

:facepalm: just after opening this issue, the flaky tests are back even with parallelism off :disappointed: So I don't think there's enough clarity about the cause to open an issue.