fix: retry logic in tests to prevent race condition failures

[yitsushi]

Closes #32

Attempt to reproduce locally

Tried to limit resources to reproduce the bug with cgroups, but I couldn't.

Setup:

cat <<-EOF > ~/.config/systemd/user/testing-slowdown.slice
[Slice]
CPUQuota=30%
MemoryHigh=800M
EOF

systemctl daemon-reload --user

Start shell (run test in that shell):

systemd-run --user --shell --slice=testing-slowdown.slice

Monitor (to make sure resources are properly throttled):

systemd-cgtop /user.slice/user-1000.slice/user@1000.service/testing.slice/testing-slowdown.slice

If I set lower limits (CPU or Memory) envtest was not able to create control plane:

2023/01/02 17:49:22 starting test env failed: unable to start control plane itself: failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)

Conclusion

I hope it fixes the issue, I'll re-run 5 times the job on this PR and see if it fails, if they are all successful, I assume that fixes the issue.

References:

• https://github.com/weaveworks/pipeline-controller/issues/32

[yitsushi]

Github Actions runs: 5/5