search

dracut-ng / dracut-ng

dracut - the event driven initramfs infrastructure
https://github.com/dracut-ng/dracut/wiki
GNU General Public License v2.0
67 stars 59 forks source link

`dracut-initramfs-restore.sh` fails on ostree-based systems #514

Open miabbott opened 1 month ago

miabbott commented 1 month ago

(Originally filed as https://github.com/dracutdevs/dracut/issues/2656)

Describe the bug

The dracut-initramfs-restore.sh script fails during dracut-shutdown.service on ostree-based systems because it incorrectly looks for the initramfs in the wrong location.

Distribution used

I've observed this on RHEL for Edge 9.4 and Fedora Silverblue 40

Dracut version

dracut-057-53.git20240104.el9.x86_64 dracut-101-1.fc40.x86_64

Init system

systemd

To Reproduce

  1. Install an ostree-based distribution
  2. Reboot
  3. Observe failures in journal

Expected behavior

The dracut-initramfs-restore.sh script should correctly find the initramfs image and unpack it to /run

Additional context

The script appears to be incorrectly identifying the initrd at /boot/efi/$MACHINE_ID/$KERNEL_VERSION/initrd, but that doesn't exist on ostree systems. However, the error does not appear to be fatal to the overall system shutdown.

On the Fedora Silverblue 40 system:

$ sudo tree /boot
/boot
├── boot -> .
├── efi
│   └── EFI
│       ├── BOOT
│       │   ├── BOOTIA32.EFI
│       │   ├── BOOTX64.EFI
│       │   ├── fbia32.efi
│       │   └── fbx64.efi
│       └── fedora
│           ├── BOOTIA32.CSV
│           ├── BOOTX64.CSV
│           ├── grub.cfg
│           ├── grubia32.efi
│           ├── grubx64.efi
│           ├── mmia32.efi
│           ├── mmx64.efi
│           ├── shim.efi
│           ├── shimia32.efi
│           └── shimx64.efi
├── grub2
│   ├── fonts
│   │   └── unicode.pf2
│   ├── grub.cfg -> ../loader/grub.cfg
│   └── grubenv
├── loader -> loader.0
├── loader.0
│   ├── entries
│   │   ├── ostree-1.conf
│   │   └── ostree-2.conf
│   └── grub.cfg
├── lost+found
└── ostree
    ├── fedora-4f6718ed80e462c0cbe6fd45efd7425235ce42e3bf59d7bf9b66b8ad1f42564a
    │   ├── initramfs-6.8.11-300.fc40.x86_64.img
    │   └── vmlinuz-6.8.11-300.fc40.x86_64
    └── fedora-80e7371f407145007e6ade4b9e47cd32c0f15acd2783d36240fbbf53da85f5eb
        ├── initramfs-6.8.11-300.fc40.x86_64.img
        └── vmlinuz-6.8.11-300.fc40.x86_64

15 directories, 24 files

On the RHEL for Edge 9.4 system:

$ sudo tree /boot
/boot
├── boot -> .
├── efi
│   └── EFI
│       ├── BOOT
│       │   ├── BOOTX64.EFI
│       │   └── fbx64.efi
│       └── redhat
│           ├── BOOTX64.CSV
│           ├── grub.cfg
│           ├── grubx64.efi
│           ├── mmx64.efi
│           ├── shim.efi
│           ├── shimx64-redhat.efi
│           └── shimx64.efi
├── grub2
│   ├── fonts
│   │   └── unicode.pf2
│   ├── grub.cfg -> ../loader/grub.cfg
│   └── grubenv
├── loader -> loader.1
├── loader.1
│   ├── entries
│   │   ├── ostree-1-rhel.conf
│   │   └── ostree-2-rhel.conf
│   └── grub.cfg
└── ostree
    ├── rhel-552aa9b24c7437e63eb8130c5777483a3f3077024ed862e8b323884248cde62d
    │   ├── initramfs-5.14.0-427.20.1.el9_4.x86_64.img
    │   └── vmlinuz-5.14.0-427.20.1.el9_4.x86_64
    └── rhel-f551d3e582fcee7e2f4b387be8e24d5e0829214ca54d74e2096ed7aac7b347f1
        ├── initramfs-5.14.0-416.el9.x86_64.img
        └── vmlinuz-5.14.0-416.el9.x86_64

13 directories, 19 files

I think the script could be changed to look for /run/ostree-booted and then unpack the initramfs.img from /usr/lib/modules/${KERNEL_VERSION}/initramfs.img

Logs from RHEL from Edge 9.4

``` $ sudo journalctl -b -1 -u dracut-shutdown [sudo] password for core: Jun 18 10:34:03 localhost systemd[1]: Starting Restore /run/initramfs on shutdown... Jun 18 10:34:03 localhost systemd[1]: Finished Restore /run/initramfs on shutdown. Jun 18 10:35:52 localhost.localdomain systemd[1]: Stopping Restore /run/initramfs on shutdown... Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1239]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1240]: gzip: stdin: unexpected end of file Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1241]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1242]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1243]: /usr/lib/dracut/dracut-initramfs-restore: line 55: bzcat: command not found Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1244]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1249]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1250]: xzcat: (stdin): File format not recognized Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1251]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1253]: /usr/lib/dracut/dracut-initramfs-restore: line 57: lz4: command not found Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1254]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1252]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1257]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1258]: /usr/lib/dracut/dracut-initramfs-restore: line 58: lzop: command not found Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1259]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1261]: /usr/lib/dracut/dracut-initramfs-restore: line 59: zstd: command not found Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1262]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1260]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1265]: ERROR: src/skipcpio/skipcpio.c:92:main(): Cannot open file '/boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd' Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1266]: cpio: premature end of archive Jun 18 10:35:52 localhost.localdomain dracut-initramfs-restore[1233]: Unpacking of /boot/efi/6db1c4c16cae4b82adac816b82eea68b/5.14.0-416.el9.x86_64/initrd to /run/initramfs failed Jun 18 10:35:52 localhost.localdomain systemd[1]: dracut-shutdown.service: Control process exited, code=exited, status=1/FAILURE Jun 18 10:35:52 localhost.localdomain systemd[1]: dracut-shutdown.service: Failed with result 'exit-code'. Jun 18 10:35:52 localhost.localdomain systemd[1]: Stopped Restore /run/initramfs on shutdown. Jun 18 10:35:52 localhost.localdomain systemd[1]: dracut-shutdown.service: Triggering OnFailure= dependencies. ```

Logs from Fedora Silverblue 40

$ sudo journalctl -b -1 -u dracut-shutdown
Jun 10 16:45:27 meatwad systemd[1]: Starting dracut-shutdown.service - Restore /run/initramfs on shutdown...
Jun 10 16:45:27 meatwad systemd[1]: Finished dracut-shutdown.service - Restore /run/initramfs on shutdown.
Jun 14 12:22:16 meatwad systemd[1]: Stopping dracut-shutdown.service - Restore /run/initramfs on shutdown...
Jun 14 12:22:16 meatwad dracut-initramfs-restore[1819558]: Unpacking of /boot/efi/20af269e759745d6b8e0def986395d5b/6.8.11-300.fc40.x86_64/initrd to /run/initramfs failed
Jun 14 12:22:16 meatwad systemd[1]: dracut-shutdown.service: Control process exited, code=exited, status=1/FAILURE
Jun 14 12:22:16 meatwad systemd[1]: dracut-shutdown.service: Failed with result 'exit-code'.
Jun 14 12:22:16 meatwad systemd[1]: Stopped dracut-shutdown.service - Restore /run/initramfs on shutdown.
Jun 14 12:22:16 meatwad systemd[1]: dracut-shutdown.service: Triggering OnFailure= dependencies.
miabbott commented 1 month ago

Comment from @jlebon that was made on original issue (https://github.com/dracutdevs/dracut/issues/2656) :

OK yeah, I thought FCOS/RHCOS didn't actually use the initramfs on shutdown. And it doesn't by default, but that changed when we started supporting iSCSI and multipathing, whose dracut modules both use need_shutdown (which is what signals that going back to the initramfs at shutdown is desirable).

So yeah... booting FCOS with multipath enabled reproduces this bug. Which means we weren't running this bit of code. Though talking with @bmarzins, the consequence of this cannot result from data loss in any way.

As for iSCSI, the module itself doesn't install a shutdown hook like the multipath one does, but maybe it's for the purpose of running the generic umount shutdown code to unmount the rootfs to ensure all data is flushed back to the target?

But yes, we should definitely fix this.