Run with `--local` and `-v /var/lib/containers/storage:/var/lib/containers/storage` failed.

[henrywang]

Run command with --local and -v /var/lib/containers/storage:/var/lib/containers/storage, e.g. sudo podman run --rm -it --privileged --pull=newer --tls-verify=false --security-opt label=type:unconfined_t -v /home/fedora/bootc-workflow-test/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --type vmdk --target-arch x86_64 --local quay.io/rhel-edge/rhel9-rhel_bootc-test:imnv

Failed with the following error:

Generating manifest manifest-vmdk.json
WARNING: target-arch is experimental and needs an installed 'qemu-user' package
panic: local storage not working, did you forget -v /var/lib/containers/storage:/var/lib/containers/storage? (not the host /var/lib/storage/containers but "/root/var/lib/containers/storage")

[cgwalters]

Yes, I hit that too. Can you test out https://github.com/osbuild/bootc-image-builder/pull/319

[cgwalters]

(Although also there is some issue where we haven’t shipped a new container for git main, so also try building current git main)

[henrywang]

Yes, I hit that too. Can you test out #319

Trying it now.

[chunfuwen]

Did this happen 100% ? and tried it without #319, I didn't have this issue.

(.libvirt-ci-venv-ci-runtest-6WVLCP) [root@dell-per730-67 libvirt-ci-latest-venv]# sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v /var/lib/libvirt/images/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --type vmdk --tls-verify=false --target-arch x86_64 --local localhost/bootc:eln --chown 107:107

....

manifest - finished successfullybuild: eb535bcdf01f8d6af1193adaa6116e09d5c0165376001914786146bd00b1c3f1 ostree-deployment: 983b9788d6b4e6f95d8c88cc09af3d5bbfc29189d8b064f5790e4b7f13dc5624 image: d480734b60dbcd95401c1c26a62bffb4d645cab20dd92961adc1541ae4d09649 qcow2: 4454f872763f6d9c4c47f196dd4211ba6c8824e558e5203a1116ce172cfe37c6 vmdk: 1d8ba1bbe77e0e6275c6a354ea421c757dba2f7200fc4e64de5eb450fc1cacf7 ovf: 62460e3ab5e28fa885fd8d09a51d506d29eefa4f45d0aa09a84bcb8c4fb8c6b1 archive: f124652b64f7d960cf9736f26e7dc486a8c266d2edc53a1fa123075fcd94fe7b Build complete! Results saved in

[chunfuwen]

it looks like issue happen on local registry tag

1)if image tag has namespace between url and repository ,for example "rhel-edge" in quay.io/rhel-edge/rhel9-rhel_bootc-test:imnv , it will throw this error.

2)If image tag don't have namespace, just like localhost/bootc:eln, it works normally

[henrywang]

@chunfuwen If the --local follows the local image localhost/image:tag, bootc switch quay.io/image:tag has to be used before upgrade.

[martinpitt]

I hit that too on today's quay.io/centos-bootc/bootc-image-builder when running it on current Fedora CoreOS. Both the host and the bootc-image-builder container have podman version 4.9.3, so that's not it. I can also run

# podman run     --rm     -it     --privileged     --security-opt label=type:unconfined_t     -v $(pwd)/config.json:/config.json     -v $(pwd)/output:/output     -v /var/lib/containers/storage:/var/lib/containers/storage --entrypoint bash    quay.io/centos-bootc/bootc-image-builder:latest 

and inside, podman images works and shows me my locally built localhost/feln-bootc:latest.

That image has the same form as the documentation's localhost/bootc:eln. But to make sure, did podman tag localhost/feln-bootc:latest localhost/bootc:eln and re-ran with literally localhost/bootc:eln, and it fails the same way.

It also fails with the unmodified container image (already pulled down locally):

podman run     --rm     -it     --privileged     --security-opt label=type:unconfined_t     -v $(pwd)/config.json:/config.json     -v $(pwd)/output:/output     -v /ostree/deploy/fedora-coreos/var/lib/containers/storage:/var/lib/containers/storage     quay.io/centos-bootc/bootc-image-builder:latest     --type qcow2     --config /config.json     --local     quay.io/centos-bootc/fedora-bootc:eln

I didn't test with #319, is there an easy way to do that? A container built from the PR or an rpm or binary to download?

Also, is there a way to just run the Go binaries? a privileged unconfined root container has no isolation anyway, and more importantly, there is a conflict potential between the podman version on the host and in the image-builder container.

I tried this:

podman create --name builder quay.io/centos-bootc/bootc-image-builder
P=$(podman mount builder)
$P/usr/bin/bootc-image-builder build --type qcow2 --store /tmp/store --config ./config.json --local localhost/feln-bootc:latest

but that fails on cp: cannot stat '/usr/bin/osbuild': No such file or directory. Yay hardcoded paths...

[martinpitt]

I tried to hack this further:

mount -o remount,rw /usr/
cp $P/usr/bin/osbuild $P/usr/bin/bootc-image-builder /usr/bin/
bootc-image-builder build --type qcow2 --store /tmp/store --config ./config.json --local localhost/feln-bootc:latest

which still fails:

Generating manifest manifest-qcow2.json
panic: local storage not working, did you forget -v /var/lib/containers/storage:/var/lib/containers/storage? (exit status 127)

so this is really not about the container bind mount or podman version incompatibilities -- but at least it is a way to test a new Go binary without an image-builder container rebuild. So is there a way to get the binaries from #139 or current main? (it's already merged)

[cgwalters]

👋 @martinpitt

I didn't test with https://github.com/osbuild/bootc-image-builder/pull/319, is there an easy way to do that? A container built from the PR or an rpm or binary to download?

bootc-image-builder is a container image. There's a Containerfile. You already know how to build it locally (or in production container build infrastructure).

(But see also https://github.com/osbuild/bootc-image-builder/tree/main/devel for some variants)

Anyways though, it seems like whatever was stuck in the Konflux build pipeline is unstuck, as

$ podman inspect --format='{{ .Created }}' quay.io/centos-bootc/bootc-image-builder:latest
2024-04-03 13:58:32.147063108 +0000 UTC

So try out the latest.

[martinpitt]

bootc-image-builder is a container image. There's a Containerfile

Whoops, of course. Thanks! :see_no_evil:

try out the latest.

It works indeed, very cool! :sunglasses: That gets rid of the expensive hack that uses a local registry.