search

osbuild / osbuild-composer

An HTTP service for building bootable OS images.
https://www.osbuild.org
Apache License 2.0
162 stars 107 forks source link

test suite should use a fixed version of cloud-image-val #2935

Open ondrejbudai opened 2 years ago

ondrejbudai commented 2 years ago

Currently, the latest tag is used, see https://github.com/osbuild/osbuild-composer/blob/7bf02771750dc7a48b752d05d04d2a45353a1845/test/cases/aws.sh#L219.

This is error-prone because any updates to cloud-image-val can cause osbuild-composer CI to start failing. This is especially true considering that cloud-image-val CI is usually red these days, see https://github.com/osbuild/cloud-image-val/commits/main Additionally, if I understand it correctly, this civ commit https://github.com/osbuild/cloud-image-val/commit/28797c0cc591284907b9a7380fa11b5f377d8dcd broke composer's CI and https://github.com/osbuild/osbuild-composer/pull/2932 was needed to fix it.

Therefore, we should just use a fixed version and update it from time to time (a bot can do it).

jrusz commented 2 years ago

What do you think about the suggestion we made with @thozza in #2932 ? Only release updated cloud-image-val container once tests on main pass there. It's using the latest osbuild-composer rpms from main for the tests there.

ondrejbudai commented 2 years ago

i'm slightly concerned about keeping the CI bits in sync between these two projects. When we add a new distro in composer, it might be missing from cloud-image-val which might cause issues in the newly added distros.

Also, using a fixed environment is how we do most stuff (the whole manifest-db concept, fixed images, fixed repositories) so it makes sense from the consistency point of view.

ghost commented 2 years ago

Using a fixed version of CIV requires a lot of manual work, specially now that CIV is getting new important tests al the time. The errors caused in rhel 9 are an exception and were caused because I'm changing the way the CI uploads images. With this PR (https://github.com/osbuild/cloud-image-val/pull/113), no changed introduced in CIV should cause an error on osbuild-composer.

The only problem I see is what you said about syncing when you introduce a new release.

ghost commented 2 years ago

With respect of CIV CI being red. This only applies to what we called published images, where all the tests apply. For IB test only a subset of tests is being executed (test without the pub tag).

We plan to get this tests green, but other important tasks were prioritized.