ginkgo tests EPIC

[Luap99]

• [ ] machine test should run in parallel
• [ ] bindings should run in parallel
• [ ] do not use os.Exit() as error handling, let ginkgo handle errors so it can provide useful output
• [ ] add test description for the matchers: Expect(...).To(..., "Description here"), this is common in the system tests. It will provide a lot more context for someone who looks at these errors.
• [ ] add linter or similar check to make sure tests use GinkgoWriter interface instead of writing to stdout/err directly

e2e tests:

• [x] check stderr for errors in cleanup (#18442)

• [x] do not use ExitWithError(), check for actual error messages (#18188)

• [x] remove flake attempts (#17967)

• [ ] remove test helpers that do not provide good error messages, i.e. GrepString/LineInOutputStartsWith/etc...

• [ ] cleanup common/setup code: there is a lot of dead and duplicated code there. We should simplify that.

• [ ] check if we can speed up slow tests, some tests run 5 seconds or more. At the bottom of the integration tests logs you have the timings of all tests. Check if there are any than can be speed up.

• [ ] remove unnecessary images from the test/initial setup, some of them are not required and can be replaced with other images. Less images means less pulls -> faster tests and less flakes

• [x] remove code duplication: BeforeEach/AfterEach blocks are duplicated in each file (#18544).

These are issues or general possible improvements I noticed while working on the ginkgo v2 migration. If you have some spare time feel free to tackle some points.

The error messages are properly to much work to change for existing tests but we definitely should require this for new tests on PR review. Running machine/bindings tests in parallel may be useful to safe some CI time/cost.

cc @containers/podman-maintainers @edsantiago

[edsantiago]

remove flake attempts (https://github.com/containers/podman/issues/17967)

I've been slamming CI on #17831. So far I've identified two tests that definitely need FlakeAttempts(3). Both are in my PR. Unfortunately there are still a bunch of other flakes that are real bugs (in tests or in podman itself or unclear). I'm whacking those when I can, and reporting the ones I can't.

[Luap99]

Yeah I don't expect this to be fixed soon, I just wanted to write that down somewhere as I think these things are important. Just properly not important enough to get prioritized.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Touching this so it stays alive. Many of the checklist items have been addressed in the past year, and the remaining ones are still worth doing.