[WIP] initoverlayfs-install: use /usr/loca/etc to conf

[dougsland]

I was able to build with osbuild but during the boot it exploded.

BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
System BootOrder not found.  Initializing defaults.
Creating boot entry "Boot0007" with label "CentOS Linux" for file "\EFI\centos\shimx64.efi"

  Booting `Automotive Stream Distribution 9 (ostree:0)'

[    1.350732] systemd[285]: Failed to execute /usr/lib/systemd/system-generators/dracut-rootfs-generator: Permission denied
[    1.367273] systemd[284]: /usr/lib/systemd/system-generators/dracut-rootfs-generator failed with exit status 1.
[    1.528011] No source specified
[    1.527933] initoverlayfs[311]: c->bootfs.val.c_str pointer is null
[    1.532968] initoverlayfs[311]: open("/boot/initoverlayfs-5.14.0-427.380.el9iv.x86_64.img", O_RDONLY| O_CLOEXEC) = -1 2 (No such file or directory)
[    1.540266] initoverlayfs[311]: losetup("(null)", "/boot/initoverlayfs-5.14.0-427.380.el9iv.x86_64.img") 2 (No such file or directory)
[    1.546895] initoverlayfs[311]: mount("(null)", "/initrofs", "erofs", MS_RDONLY, NULL) 22 (Invalid argument)
[    1.552621] initoverlayfs[311]: mount("/boot", "/initoverlayfs/boot", "(null)", MS_MOVE, NULL) 2 (No such file or directory)
[FAILED] Failed to start Switch Root pre-initoverlayfs.
[    2.658479] overlayfs: empty lowerdir
[    7.830376] No source specified
[    7.830543] overlayfs: overlay with incompat feature 'volatile' cannot be mounted
[    7.830781] i[    7.830799] overlayfs: overlay with incompat feature 'volatile' cannot be mounted
nitoverlayfs[375]: c->bootfs.val.c_str pointer is null
[    7.831903] initoverlayfs[375]: open("/boot/initoverlayfs-5.14.0-427.380.el9iv.x86_64.img", O_RDONLY| O_CLOEXEC) = -1 2 (No such file or directory)
[    7.840708] initoverlayfs[375]: losetup("(null)", "/boot/initoverlayfs-5.14.0-427.380.el9iv.x86_64.img") 2 (No such file or directory)
[    7.841880] initoverlayfs[375]: mount("(null)", "/[FAILED] Failed unmounting /sysroot/sysroot.
[FAILED] Failed unmounting /sysroot.
[FAILED] Failed to start Switch Root pre-initoverlayfs.

[ericcurtin]

You should maybe change this:

conf_read(&conf, "/etc/initoverlayfs.conf");

to this:

conf_read(&conf, "/etc/local/initoverlayfs.conf");

I haven't tested or anything though... Maybe there's a couple of things in different places...

This has also pointed out that we probably need more logging inside conf_read. Like log when fopen fails.

[ericcurtin]

Oh wait scratch that though maybe not because you copied to:

/etc/initoverlayfs.conf

In initrd...

Anyway, it looks like bootfs variable is empty, that comes from /etc/initoverlayfs.conf file

[ericcurtin]

It's the variable that contains bootfs details, so it knows which partition to mount for /boot

[ericcurtin]

We should be very close here, we just need to produce more output in osbuild to see where this is falling over. Could you paste the full output of the dracut section, including the names of the osbuild stages that are running dracut @dougsland ?

[dougsland]

We should be very close here, we just need to produce more output in osbuild to see where this is falling over. Could you >paste the full output of the dracut section, including the names of the osbuild stages that are running dracut @dougsland ?

Still failing for reach /usr/local/etc/ using inst_simple /usr/local/etc/initoverlayfs.conf /etc/initoverlayfs.conf or inst_dir /boot /initrofs /overlay /overlay/upper /overlay/work /usr /usr/local/ /usr/local/etc/

dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: initoverlayfs ***
dracut-install: ERROR: installing '/usr/local'
dracut-install: ERROR: failed to create directory '/tmp/dracut/dracut.NtKL7s/initramfs/usr/local'
dracut-install: ERROR: installing '/usr/local/etc'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /tmp/dracut/dracut.NtKL7s/initramfs -d /boot /initrofs /overlay /overlay/upper /overlay/work /usr /usr/local /usr/local/etc

However, dracut output seems more friendly when using /initoverlayfs.conf:

Creating empty machine-id
/usr/bin/dracut: line 1064: /sys/module/firmware_class/parameters/path: No such file or directory
dracut: Executing: /usr/bin/dracut --reproducible -v --add ostree --tmpdir=/tmp/dracut -f /tmp/initramfs.img --no-hostonly --kver 5.14.0-432.384.el9iv.x86_64
dracut: dracut module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found!
dracut: dracut module 'systemd-resolved' will not be installed, because command '/usr/lib/systemd/systemd-resolved' could not be found!
dracut: dracut module 'systemd-timesyncd' will not be installed, because command '/usr/lib/systemd/systemd-timesyncd' could not be found!
dracut: dracut module 'systemd-timesyncd' will not be installed, because command '/usr/lib/systemd/systemd-time-wait-sync' could not be found!
dracut: dracut module 'modsign' will not be installed, because command 'keyctl' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-daemon' will not be installed, because command 'dbus-daemon' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: 62bluetooth: Could not find any command of '/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'!
dracut: dracut module 'lvmmerge' will not be installed, because command 'lvm' could not be found!
dracut: dracut module 'lvmthinpool-monitor' will not be installed, because command 'lvm' could not be found!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'crypt' depends on 'dm', which can't be installed
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'lvm' will not be installed, because command 'lvm' could not be found!
dracut: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut: dracut module 'pcsc' will not be installed, because command 'pcscd' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: initoverlayfs ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: ostree ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: microcode_ctl-fw_dir_override ***
/usr/lib/dracut/modules.d/99microcode_ctl-fw_dir_override/module-setup.sh: line 16: /sys/module/firmware_class/parameters/path: No such file or directory
dracut:   microcode_ctl module: mangling fw_dir
dracut:     microcode_ctl: reset fw_dir to "/lib/firmware/updates /lib/firmware"
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel"...
intel: model '', path ' intel-ucode/*', kvers ''
dracut:       microcode_ctl: intel: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07"...
intel-06-2d-07: model 'GenuineIntel 06-2d-07', path ' intel-ucode/06-2d-07', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-2d-07: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4e-03"...
intel-06-4e-03: model 'GenuineIntel 06-4e-03', path ' intel-ucode/06-4e-03', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-4e-03: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "5.14.0-432.384.el9iv.x86_64" failed early load check for "intel-06-4e-03", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4f-01"...
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.17.0 3.10.0-894 3.10.0-862.6.1 3.10.0-693.35.1 3.10.0-514.52.1 3.10.0-327.70.1 2.6.32-754.1.1 2.6.32-573.58.1 2.6.32-504.71.1 2.6.32-431.90.1 2.6.32-358.90.1'
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-4f-01: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "5.14.0-432.384.el9iv.x86_64" failed early load check for "intel-06-4f-01", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04"...
intel-06-55-04: model 'GenuineIntel 06-55-04', path ' intel-ucode/06-55-04', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-55-04: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03"...
intel-06-5e-03: model 'GenuineIntel 06-5e-03', path ' intel-ucode/06-5e-03', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-5e-03: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01"...
intel-06-8c-01: model 'GenuineIntel 06-8c-01', path ' intel-ucode/06-8c-01', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-8c-01: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-0xca"...
intel-06-8e-9e-0x-0xca: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
intel-06-8e-9e-0x-0xca: caveat is disabled in configuration
dracut:     microcode_ctl: kernel version "5.14.0-432.384.el9iv.x86_64" failed early load check for "intel-06-8e-9e-0x-0xca", skipping
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell"...
intel-06-8e-9e-0x-dell: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel: calling check_caveat 'intel' '1' match_model=0
intel: model '', path ' intel-ucode/*', kvers ''
Dependency check for required intel succeeded: result=0
dracut:       microcode_ctl: intel-06-8e-9e-0x-dell: caveats check for kernel version "5.14.0-432.384.el9iv.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell" to fw_dir variable
dracut:     microcode_ctl: final fw_dir: "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-8e-9e-0x-dell /usr/share/microcode_ctl/ucode_with_caveats/intel-06-8c-01 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04 /usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07 /usr/share/microcode_ctl/ucode_with_caveats/intel /lib/firmware/updates /lib/firmware"
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:           real
dracut: Files:          282
dracut: Linked:         4 files
dracut: Compared:       0 xattrs
dracut: Compared:       10 files
dracut: Saved:          1.04 MiB
dracut: Duration:       0.004054 seconds
dracut: *** Hardlinking files done ***
dracut: Could not find 'strip'. Not stripping the initramfs.
dracut: *** Store current command line parameters ***
dracut: *** Creating image file '/tmp/initramfs.img' ***
dracut: *** Creating initramfs image file '/tmp/initramfs.img' done ***

but when booting:

BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
System BootOrder not found.  Initializing defaults.
Creating boot entry "Boot0007" with label "CentOS Linux" for file "\EFI\centos\shimx64.efi"

  Booting `Automotive Stream Distribution 9 (ostree:0)'

[    1.292355] systemd[293]: Failed to execute /usr/lib/systemd/system-generators/dracut-rootfs-generator: Permission denied
[    1.318070] systemd[292]: /usr/lib/systemd/system-generators/dracut-rootfs-generator failed with exit status 1.
[FAILED] Failed to start pre-initoverlayfs initialization.
[FAILED] Failed to start Switch Root pre-initoverlayfs.
[    2.563812] overlayfs: empty lowerdir
[FAILED] Failed unmounting /sysroot/sysroot.
[FAILED] Failed unmounting /sysroot.
[FAILED] Failed to start Switch Root pre-initoverlayfs.