bound images fail in bootc-image-builder

[cgwalters]

time="2024-07-22T15:53:02Z" level=error msg="finding config on system: unable to resolve HOME directory: user: lookup userid 0: no such file or directory"

``` org.osbuild.bootc.install-to-filesystem: f60d96bf6cf76c88a936d2ca78536662205cc603ca2a67e6b1681126c7190443 { "kernel-args": [ "rw", "console=tty0", "console=ttyS0" ], "target-imgref": "quay.io/cgwalters/ostest:bound" } device/disk (org.osbuild.loopback): loop5 acquired (locked: False) mount/- (org.osbuild.xfs): mounting /dev/loop5p4 -> /store/tmp/buildroot-tmp-vk85an87/mounts/ mount/boot (org.osbuild.ext4): mounting /dev/loop5p3 -> /store/tmp/buildroot-tmp-vk85an87/mounts/boot mount/boot-efi (org.osbuild.fat): mounting /dev/loop5p2 -> /store/tmp/buildroot-tmp-vk85an87/mounts/boot/efi Host kernel does not have SELinux support, but target enables it by default; this is less well tested. See https://github.com/containers/bootc/issues/419 Installing image: docker://quay.io/cgwalters/ostest:bound Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot layers already present: 0; layers needed: 76 (1.7 GB) Fetched layers: 1.61 GiB in 18 seconds (91.60 MiB/s) Deploying container image...Freed objects: 118 bytes done Running bootupctl to install bootloader > bootupctl backend install --write-uuid --device /dev/loop5 /run/osbuild/mounts Installed: grub.cfg Installed: "centos/grub.cfg" Mounting deployment /var Initializing /var/tmp > systemd-tmpfiles --create --boot --prefix=/var/tmp /usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to replace specifiers in '/var/log/journal/%m': No such file or directory /usr/lib/tmpfiles.d/systemd.conf:23: Failed to replace specifiers in '/run/log/journal/%m': No such file or directory Failed to parse ACL "d:group::r-x,d:group:adm:r-x,d:group:wheel:r-x,group::r-x,group:adm:r-x,group:wheel:r-x": No such file or directory. Ignoring /usr/lib/tmpfiles.d/systemd.conf:25: Failed to replace specifiers in '/run/log/journal/%m': No such file or directory /usr/lib/tmpfiles.d/systemd.conf:26: Failed to replace specifiers in '/run/log/journal/%m/*.journal*': No such file or directory /usr/lib/tmpfiles.d/systemd.conf:29: Failed to replace specifiers in '/var/log/journal/%m': No such file or directory /usr/lib/tmpfiles.d/systemd.conf:30: Failed to replace specifiers in '/var/log/journal/%m/system.journal': No such file or directory Failed to parse ACL "d:group::r-x,d:group:adm:r-x,d:group:wheel:r-x,group::r-x,group:adm:r-x,group:wheel:r-x": No such file or directory. Ignoring /usr/lib/tmpfiles.d/systemd.conf:32: Failed to replace specifiers in '/var/log/journal/%m': No such file or directory /usr/lib/tmpfiles.d/systemd.conf:33: Failed to replace specifiers in '/var/log/journal/%m/system.journal': No such file or directory Failed to parse ACL "default:group:tss:rwx": No such file or directory. Ignoring Failed to parse ACL "default:group:tss:rwx": No such file or directory. Ignoring Pulling bound image time="2024-07-22T15:53:02Z" level=error msg="finding config on system: unable to resolve HOME directory: user: lookup userid 0: no such file or directory" ERROR Installing to filesystem: pull bound images: Task Pulling bound image failed: ExitStatus(unix_wait_status(256)) Traceback (most recent call last): File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in r = main(args["options"], args["inputs"], args["paths"]) File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main subprocess.run(pargs, env=env, check=True) File "/usr/lib64/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]73f563df6d8472b839a06b3f3686efb905289d24e4d42ee6cc76b29742c0a08c', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'quay.io/cgwalters/ostest:bound', '/run/osbuild/mounts']' returned non-zero exit status 1. mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-vk85an87/mounts/boot/efi unmounted mount/boot (org.osbuild.ext4): umount: /store/tmp/buildroot-tmp-vk85an87/mounts/boot unmounted mount/- (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-vk85an87/mounts/ unmounted ```

I previously hit https://github.com/containers/podman/issues/22785#issuecomment-2131384748 - this looks like a different one.

Will dig in to see what's going on.

[cgwalters]

Can someone remind me of the best way to get a debug shell in an osbuild pipeline in bib? cc @mvo5

[cgwalters]

OK yep there's no /etc/passwd in osbuild stages, or at least not in the stage running bootc install:

ls: cannot access '/etc/passwd': No such file or directory

[cgwalters]

https://github.com/containers/bootc/pull/716 Next up is Error: mkdir /etc/containers/networks: read-only file system as expected

[cgwalters]

Next up is Error: mkdir /etc/containers/networks: read-only file system as expected :arrow_right: https://github.com/containers/bootc/pull/717

And next up is: Error: initializing source docker://quay.io/fedora/fedora:40: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on [::1]:53: read udp [::1]:54423->[::1]:53: read: connection refuse

Because it looks like bootc-image-builder is running stages without networking...as expected.

Now, this gets into an important implementation detail. Today bib copies the input container to the osbuild store...I want to avoid that for these bound images as it complicates things and we are not getting value out of the osbuild store today.

IOW I want to just require that bound images live in the container storage used by bib - as that will be maximally erognomic for users.

So to fix this, we'll want to configure the install path invocation to look there as an additional image store or oso.

[cgwalters]

https://github.com/containers/bootc/pull/719 will get us closer - I didn't yet get it working end-to-end in bib, but it does work with bootc install in the way I think it should.

[cgwalters]

Filed https://github.com/osbuild/bootc-image-builder/issues/560 for the bib side discussion