initoverlayfs is failing to boot after install on c9s

[Yarboa]

Working on this minimal packit test #28, seems that fedora is working with initoverlayfs, but c9s machine fails to boot.

distro: CentOS Stream 9 kernel: 5.14.0-388.el9.x86_64

https://github.com/containers/initoverlayfs/pull/28/checks?check_run_id=19189680972

Need to debug manually for more details

[ericcurtin]

The zero logs from the boot may make this difficult but lets see, I may have a fix that works regardless

[Yarboa]

@ecurtin Adding logs https://pastebin.com/c4nL16WJ line #573

initoverlay-install output backup https://paste.centos.org/view/b2678437

[Yarboa]

Will propose different PR for c9s

Do not forget to add this one https://github.com/containers/initoverlayfs/pull/28#discussion_r1413771812

[ecurtin]

@Yarboa Think you meant to tag @ericcurtin , unless you really want this random ML Ops Engineer to parachute in from deep space and have opinions about C.

[ericcurtin]

Sorry @ecurtin ecurtin is my username on many other things :smile:

[Yarboa]

@ericcurtin Anyway Attaching this Log It seems that first init, does not read storage properly https://pastebin.com/jn8gU0Y8

With the following error, I could not point on a difference between c9s and fedora related to this ? https://github.com/containers/initoverlayfs/blob/main/storage-init.c#L594

[    1.148667] storage-init: bootfs: {"", "bootfs "}, bootfstype: {"ext4", "bootfstype ext4"}, fs: {"(null)", "(null)"}, fstype: {"(null)", "(null)"}, udev_trigger: {"udevadm trigger --type=devices --action=add --subsystem-match=module --subsystem-match=block --subsystem-match=virtio --subsystem-match=pci --subsystem-match=nvme --subsystem-match=mmc --subsystem-match=mmc_host --subsystem-match=platform", "udev_trigger udevadm trigger --type=devices --action=add --subsystem-match=module --subsystem-match=block --subsystem-match=virtio --subsystem-match=pci --subsystem-match=nvme --subsystem-match=mmc --subsystem-match=mmc_host --subsystem-match=platform"}, udev_trigger_generic: {"udevadm trigger --type=devices --action=add", "udev_trigger_generic udevadm trigger --type=devices --action=add"}
Starting systemd-udevd version 252-18.el9
[    1.157468] storage-init: fork_execvp_no_wait(0x17c2880)
[    1.157649] storage-init: forked 209 fork_execvp_no_wait
[    1.157652] storage-init: c->bootfs.val.c_str string is ""
[    1.160622] loop: module loaded
[    1.171546] storage-init: fork_execlp_no_wait("udevadm")
[    1.171598] storage-init: forked 214 fork_execlp_no_wait
Device path cannot contain "..".
[    1.188393] storage-init: optimized udev trigger failed, fall back to generic: 1 && 1
[    1.188400] storage-init: fork_execvp_no_wait(0x17c29c0)
[    1.188474] storage-init: forked 225 fork_execvp_no_wait
[    1.199644] virtio_blk virtio2: 2/0/0 default/read/poll queues
[    1.201613] virtio_blk virtio2: [vda] 20971520 512-byte logical blocks (10.7 GB/10.0 GiB)
[    1.206224]  vda: vda1 vda2 vda3
[    1.230533] storage-init: fork_execlp("udevadm")
Device path cannot contain "..".