search

dracutdevs / dracut

dracut the event driven initramfs infrastructure
https://github.com/dracutdevs/dracut/wiki
GNU General Public License v2.0
599 stars 399 forks source link

rd.live.overlay.overlayfs=1 is broken on Fedora 39 LiveOS #2645

Open mahdiaqallal opened 5 months ago

mahdiaqallal commented 5 months ago

Describe the bug A clear and concise description of what the error is.

rd.live.overlay.overlayfs=1 is supposed to provide a non-persistent (i.e. deleted after reboot of Live Image) and temporary storage in RAM at /run/overlayfs which is by default 32GiB unless another parameter is supplied : rd.live.overlay.size=<size_MiB> . See dracut documentation #Booting live images

Using rd.live.overlay.overlayfs=1 as kernel command line parameter has two issues

  1. sudo dmesg | grep overlayfs includes an error: [ 13.317280] overlayfs: failed to resolve '/run/overlayfs': -2

  2. Installing any rpm package once Fedora 39 LiveOS is running, results with the same errors error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied error: Plugin selinux: hook fsm_file_prepare failed [...] Error unpacking rpm package Failed: Error: Transaction failed

Below an example with openssl

 sudo dnf install -y openssl`
Last metadata expiration check: 1:44:39 ago on Sun 07 Apr 2024 08:41:58 PM CEST.
Dependencies resolved.
===============================================================================
 Package                  Architecture            Version                            Repository               Size
===============================================================================
Installing:
 openssl                  x86_64                  1:3.1.1-4.fc39                     fedora                  1.0 M

Transaction Summary
===============================================================================
Install  1 Package

Total download size: 1.0 M
Installed size: 1.6 M
Downloading Packages:
openssl-3.1.1-4.fc39.x86_64.rpm                                                    1.4 MB/s | 1.0 MB     00:00    
-------------------------------------------------------------------------------------------------------------------
Total                                                                              802 kB/s | 1.0 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                           1/1 
  Installing       : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 
error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied
error: Plugin selinux: hook fsm_file_prepare failed

Error unpacking rpm package openssl-1:3.1.1-4.fc39.x86_64
  Verifying        : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 

Failed:
  openssl-1:3.1.1-4.fc39.x86_64                                                                                    

Error: Transaction failed

Distribution used Which distribution was this behaviour seen in? Fedora 39 LiveOS

Dracut version Which dracut version was this behaviour seen in? Fedora 39 dracut-059-16.fc39 Init system Which init system is being used? systemd To Reproduce Steps or code to reproduce the behavior. Insert alongside your kernel command line parameter on Fedora 39 Live ISO (for example in a grub.cfg menuentry) rd.live.overlay.overlayfs=1 after the parameter rd.live.image Expected behavior A clear and concise description of what you expected to happen.

We should have with this parameter :

  1. a root live filesystem with 32GiB of available space instead of 4.9G is used and 1.5G is available space out of a total size of 6.4G

  2. the ability to install rpm packages with sudo dnf install <package> until the available RAM is depleted

Additional context Add any other context you like about the problem here.

a. Users of Fedora Linux have been reporting related bugs here: https://discussion.fedoraproject.org/t/fedora-liveos-root-system-and-available-ram/82531/1

CC @gregory-lee-bartholomew

(Cherry-picked commit from https://github.com/dracutdevs/dracut/pull/2604)

b. The Booting Live Images documentation section could need be more "verbose" with more explanations, examples, use-cases. IMHO

mahdiaqallal commented 5 months ago

Food for thoughts: couldn't it be related with SELinux rules because of the folders where /run/overlayfs stores all modified files and directories after thoses unpacked from squashfs in the live boot ?

I found this thread in Fedora Discussion and this and that bugzilla report

Can't be sure if it helps or not, it is well beyond my understanding

I didn't use any of the livecd-tools to reproduce this issue. The Fedora 39 LiveOS was directly written into an usb using following command sudo dd if=/<path-to-Fedora 39 LiveOS.iso> of=/dev/sdb

Then the usb drive was booted and it's grub.cfg modified according to my description above

FYI: livecd-tools can't be used as of Fedora 39 and since Fedora 35+ because the structure of the .iso files has been changed and an appropriate release still needs to be available from the official Fedora repos. see this issues https://github.com/livecd-tools/livecd-tools/issues/244 and https://github.com/livecd-tools/livecd-tools/issues/253 in livecd-tools repository

FGrose commented 5 months ago

The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso Those problems are not related to the rd.live.overlay.overlayfs option.

The dracut documentation #Booting live images is incomplete. It should rather say:

SquashFS (read-only) base filesystem image Note — There are 3 separate overlay types available:

Using one of these technologies, the system will provide a writable overlay for the base, read-only SquashFS root filesystem. These methods enable a relatively fast boot and lower RAM usage.

With the original Device-mapper snapshot overlay, users must be careful to avoid writing too many blocks to the snapshot device. Once the blocks of the snapshot overlay are exhausted, the whole root filesystem becomes read-only leading to application failures. The snapshot overlay device is marked 'Overflow', and a difficult recovery is required to repair and enlarge the overlay offline.

When rd.live.overlay= is not specified for persistent overlay storage, or the specified file is not found or writable, a Device-mapper snapshot based non-persistent or temporary overlay is automatically created as a sparse file in RAM of the initramfs. This file will only consume content space as required blocks are allocated. This snapshot based overlay defaults to an apparent size of 32 GiB in RAM, and can be adjusted with the rd.live.overlay.size= kernel command line option. This file is hidden (and appears deleted) when the boot process switches out of the initramfs to the main root filesystem but its loop device remains connected to the Device-mapper snapshot.

Even with large Device-mapper overlay files for write space, the available root filesystem capacity is limited by the total allocated size of the base root filesystem, which often provide only a small number of gigabytes of free space.

This shortage could be remedied by building the root filesystem with more allocated free space, or the OverlayFS based overlay mount method can be used.

When the rd.live.overlay.overlayfs option is specified or when rd.live.overlay= points to an appropriate directory with a sister at /../ovlwork, then an OverlayFS based overlay mount is employed. Such a persistent OverlayFS overlay can extend the available root filesystem storage up to the capacity of the LiveOS disk device.

For non-persistent OverlayFS overlays, the /run/overlayfs directory in the /run tmpfs is used for temporary storage. This filesystem is typically sized to one half of the RAM total in the system. The command: mount -o remount,size=<nbytes> /run will resize this virtual filesystem after booting.

Cross reference link https://github.com/dracut-ng/dracut-ng/issues/137

mahdiaqallal commented 5 months ago

@FGrose : thanks for your reply . It might need some elaboration though:

Which kernel command line parameter is needed to activate on a Fedora Worsktation Live .ISO, the "Device-mapper snapshot based non-persistent overlays":

Device-mapper snapshot based non-persistent overlays are sparse files in RAM that only consume content space as required blocks are allocated. These snapshot based overlays default to an apparent size of 32 GiB in RAM.

Regarding live-respins

The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso

Are live-respins natively corrected for rd.live.overlay.overlayfs=1 ? Does this mean that on Fedora 40 Worsktation Live ISO the bug for rd.live.overlay.overlayfs=1 is fixed too ?

I'll try soon sudo mount -o remount,size=70% /run (on 32GiB RAM laptop) on Fedora 40 Workstation LIve ISO and will provide feedback

mahdiaqallal commented 5 months ago

@FGrose Thanks for updating the man page. Please it needs some enhancement like examples for the two use cases below so I can test it on Fedora 40beta Workstation LIve ISO command line parameters and provide feedback

What are the minimal kernel command line parameters to activate (on a freshly created Fedora 40beta Workstation Live USB):

  1. Temporary in memory ove OR
  2. Permanent boot reristant on disk overlay (provided a dedicated vfat or ext4 or btrfs partition has been created)