Feedback on running osbuild in GitLab CI for CKI project

[mh21]

Purpose was to build gitlab-runner worker VM images which are basically FC37 + moby similar to https://github.com/major/imagebuilder-fedora/.

The setup is a service container image with osbuild in it, which can then be accessed in GitLab via a socket in /builds.

• minimal example: https://gitlab.com/cki-project/experimental/osbuild-example

main issues

• ~GitLab-provided runners use a 5.4+ kernel which is missing LOOP_CONFIGURE support (job logs); this can be worked around by using self-provisioned runners with a newer kernel~: fixed by https://github.com/osbuild/osbuild/pull/1253
• FC37+ doesn't have the loop devices preallocated, while Ubuntu used in the blog post above does that; as devices are not synced after container creation, osbuild is unhappy because it cannot find its loop devices (job logs); this can be worked around with a systemd unit that takes care of creating the devices
• osbuild-composer only exposes a local socket in /run for the use by composer-cli; this can be worked around via systemd-socket-proxyd rerouting to a socket in the shared /builds volume and --socket

misc issues

• looking at the volumes, they are gp2, is that intended given all the discussions about gp3 being faster and cheaper?
• https://github.com/osbuild/osbuild-composer/issues/1170 is a bit painful on a shared account
• looking at the API calls at https://github.com/osbuild/osbuild-composer/blob/main/internal/cloud/awscloud/awscloud.go, it seems to be a bit too easy to end up with (unnamed/untagged) dangling snapshots or AMIs if a job gets killed somewhere in the middle
• a --wait parameter is missing that would simplify CLI scripts
• there is no support to pass local environment variables like AWS_DEFAULT_REGION/AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY into the API calls, e.g. via something like --pass-aws-environment-variables
• there is no CLI support for passing tags for the snapshots/ami that are created
• it would be nice if the socket path could be taken from a WELDR_API_SOCKET environment variable
• the ID of the created ami is not provided via composer-cli --json compose info
• having both https://weldr.io/lorax/composer-cli.html (outdated) and https://www.osbuild.org/guides/image-builder-on-premises/building-an-image-from-cli.html (new and shiny) is confusing
• the required AWS policies for image uploading/importing are not documented

[mh21]

/cc @ondrejbudai

[AnthonyAus]

/cc @mh21 I'm trying composer in a container, and getting setfile errors during the compose build: ... raise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command '['setfiles', '-F', '-r', '/run/osbuild/tree', '/run/osbuild/tree/etc/selinux/targeted/contexts/files/file_contexts', '/run/osbuild/tree']' returned non-zero exit status 255.\n"}]},"metadata":null},"upload_status":"failure","pipeline_names":{"build":["build"],"payload":["os","image","vmdk"]},"host_os":"rhel-92","arch":"x86_64","image_boot_mode":"hybrid","job_error":{"id":10,"reason":"osbuild build failed"}},"channel":"","queued_at":"2024-01-16T21:34:47.325360649Z","started_at":"2024-01-16T21:34:47.33913667Z","finished_at":"2024-01-16T21:35:40.764334525Z","expires_at":"0001-01-01T00:00:00Z","retries":0}

Are you able to advise tips on getting setfiles working in the container, I understand we can't enable selinux inside a podman container

[mh21]

no 😕