Filetrigger failure during mkosi image build

[Werkov]

I use mkosi when hacking systemd (in order to build an image to run in qemu/systemd-nspawn).

My build fails at command:

 "bwrap --dev-bind / / --chdir /home/mkoutny/projects/systemd-pidstore --tmpfs /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/run --tmpfs /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/tmp --bind /home/mkoutny/.cache/mkosi-workspacez4uq8ss3 /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/var/tmp --proc /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/proc --dev /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/dev --ro-bind /sys /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/sys --unsetenv TMPDIR --ro-bind /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/etc/machine-id /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/etc/machine-id --bind /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/etc/passwd /etc/passwd --bind /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/etc/group /etc/group --bind /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/etc/shadow /etc/shadow --bind /dev/null /etc/gshadow sh -c 'chmod 1777 /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/tmp /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/var/tmp /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/dev/shm && mkdir /home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/run/host && echo mkosi >/home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root/run/host/container-manager && exec $0 "$@"' env ZYPP_CONF=/home/mkoutny/.cache/mkosi-workspacez4uq8ss3/pkgmngr/etc/zypp/zypp.conf \
 zypper --root=/home/mkoutny/.cache/mkosi-workspacez4uq8ss3/root --cache-dir=/home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp --reposd-dir=/home/mkoutny/.cache/mkosi-workspacez4uq8ss3/pkgmngr/etc/zypp/repos.d --gpg-auto-import-keys --non-interactive \
 install --download in-advance acl bash-completion bpftool btrfs-progs \
 coreutils cryptsetup dbus-broker diffutils dnsmasq dosfstools e2fsprogs \
 f2fs-tools findutils gcc gdb glibc-locale-base grep gzip kbd kernel-kvmsmall \
 kexec-tools less libcap-ng-utils mtools nano nftables openssh-server openssl \
 python3 python3-pefile python3-psutil python3-pytest qrencode quota sed shadow \
 socat strace systemd tar tmux tree udev util-linux valgrind vim \
 wireguard-tools xfsprogs zsh" \
 returned non-zero exit code 107.

Exit code 107 - ZYPPER_EXIT_INF_RPM_SCRIPT_FAILED points me to

(126/146) Installing: kexec-tools-2.0.27-2.2.x86_64 ........................................................................................................................................................................................................................[done]
warning: /home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp/packages/repo-oss/x86_64/sdbootutil-1+git20231214.b186b2d-1.1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY
Error: No ESP detected. Legacy system?
warning: %transfiletriggerin(sdbootutil-1+git20231214.b186b2d-1.1.x86_64) scriptlet failed, exit status 1
(127/146) Installing: sdbootutil-1+git20231214.b186b2d-1.1.x86_64 ..........................................................................................................................................................................................................[done]
warning: /home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp/packages/repo-oss/noarch/python311-zipp-3.17.0-1.1.noarch.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY

I believe a change in 6bcf1d3..b186b2d caused the issue since the image with former version of sdbootutil builds fine. It points to the newly added transfiletriggerin (in 8f4c552c274301341399605048ac1a68cb3c22f0).

I can't (b)wrap my head around it, more questions than answers:

• Why does transfiletriggerin apparently run (and fails) when the package itself is installed? (without triggering paths)
• Why does it fail with missing ESP, when ESP correctly lists $BOOT=?
  • manual bootctl inside the image works outputs no $BOOT= line in "Boot Loader Entries" section, this is during the build process, not inside qemu, the invocation on the host lists the $BOOT= line
  • manual invocation sdbootutil update fails with "Error: Can't determine root subvolume" though (I may have been in a wrong "container")
• What pulls sdbootutil into the image actually? (So that I could work this around.)

Interestingly, systemd Github CI succeeds (the difference is that the build-host is Ubuntu, not openSUSE Tumbleweed (my host) and it uses dnf instead of zypper to roll out RPMs).

[lnussel]

you can export YAST_IS_RUNNING=instsys when building images to avoid running certain scriptlets. If you want the package to update systemd-boot, you can set SYSTEMD_ESP_PATH

[Werkov]

Thanks, the YAST_IS_RUNNING variable does the trick (on my local invocation).

mkosi builds UKI images out of band and systemd-boot picks them. So I don't necessarily need the explicit update.

However, to have a smooth install, I tried specifying SYSTEMD_ESP_PATH inside the build container, however, it would fail at check:

170/173) Installing: dracut-059+suse.533.g5a7cf9fa-1.1.x86_64 .............................................................................................................................................................................................................[done]
warning: /home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp/packages/repo-oss/x86_64/sdbootutil-1+git20231214.b186b2d-1.1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY
Error: mismatch of esp path
warning: %transfiletriggerin(sdbootutil-1+git20231214.b186b2d-1.1.x86_64) scriptlet failed, exit status 1
(171/173) Installing: sdbootutil-1+git20231214.b186b2d-1.1.x86_64 ..........................................................................................................................................................................................................[done]
warning: /home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp/packages/repo-oss/x86_64/sdbootutil-rpm-scriptlets-1+git20231214.b186b2d-1.1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY
(172/173) Installing: sdbootutil-rpm-scriptlets-1+git20231214.b186b2d-1.1.x86_64 ...........................................................................................................................................................................................[done]
warning: /home/mkoutny/projects/systemd-pidstore/mkosi.cache/zypp/packages/repo-oss/x86_64/kernel-kvmsmall-6.6.7-1.1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY
Error: mismatch of esp path
Error: mismatch of esp path
(173/173) Installing: kernel-kvmsmall-6.6.7-1.1.x86_64 .....................................................................................................................................................................................................................[done]

That check seems too strict inside image builds.

[Werkov]

Also, the zypper output looks like the triggers when triggered by kernel-kvmsmall won't cause a non-zero exit code, while it is somehow triggered at installation of sdbootutil where it apparently propagates into the exit code.

Could this be zypper triggers bug afterall?

[lnussel]

Not sure how that error message happens. bootctl is supposed to use SYSTEMD_ESP_PATH and sdbootutil checks whether it actually reports that path in it's output. If bootctl doesn't honor SYSTEMD_ESP_PATH it means the path e.g. doesn't exist. I'd expect an error message in that case though: https://github.com/systemd/systemd/blob/main/src/shared/find-esp.c#L489