search

getsops / sops

Simple and flexible tool for managing secrets
https://getsops.io/
Mozilla Public License 2.0
16.97k stars 878 forks source link

CI: the same tests are run five times #1292

Open felixfontein opened 1 year ago

felixfontein commented 1 year ago

While looking a bit closer at CI I noticed that make test is run in all five Build and test ${{ matrix.os }} ${{ matrix.arch }} GHA runs. Since every of these runs runs on ubuntu-latest, the exact same tests are run in all five runs. This seems a bit wasteful, especially since running these tests takes around 2/3 of the total run time for each of these runs.

I would propose to run the unit tests (make test) only once in a second stage, similar to the functional tests - or maybe as an intermediate stage between build and functional tests?

@getsops/maintainers what do you think?

hiddeco commented 1 year ago

Likely a bug I introduced, and probably should look more like: https://github.com/hiddeco/sshsig/blob/main/.github/workflows/test.yaml#L18

felixfontein commented 1 year ago

It's nothing you introduced, it has been like that for a longer time: a115ce479f7d3e05a10ce2cedb6e59c2a5a17efd You only increased the amount of parallel runs from 3 to 5 ;-)

I checked back the travis config, https://github.com/getsops/sops/blob/886c1ee78144be6328f546d3aae09445562aefff/.travis.yml#L28, it simply ran them once and all on the same platform (probably some form of Linux).

hiddeco commented 1 year ago

Looking into this, the better way would be to leverage QEMU in combination with binfmt_misc (1, 2) while running e.g. ARM64 tests with GOARCH set (3).

In combination with the available Ubuntu, MacOS and Windows GitHub runners, this would allow us to run the Go tests for almost every platform (except for Darwin/ARM64). While running integration tests for at least every platform (but not every CPU architecture).