init exits with an exitcode 0x00000200

[0n0w1c]

My first attempts to use Booster have ended with the following: https://forum.manjaro.org/uploads/default/original/3X/2/9/298163cc84c62c56803c0800295656180475e444.jpeg

Init exits with an exitcode 0x00000200

$ cat grub.cfg (clipped)

menuentry "Manjaro Booster Test" {
          echo "Manjaro Booster Test booting..."
          insmod part_gpt
          insmod btrfs
          linux (hd0,gpt3)/@/boot/kernel8.img root=UUID=da0a03c1-016b-4782-824e-fe85ac1642c7 rw rootflags=defaults,ssd,compress-force=zstd,subvol=@ rootwait snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=0 console=serial0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=serial0,115200 audit=0
          initrd (hd0,gpt3)/@/boot/booster.img
          boot
}

$ cat /etc/booster.yaml 
network:
   interfaces: eth0
   dhcpd: on
universal: false
modules: -*,hid_apple,kernel/fs/btrfs/btrfs.ko,kernel/drivers/block/zram/zram.ko
compression: zstd
strip: false
extra_files: mount,fsck,fsck.btrfs,btrfsck,btrfs,zramctl
vconsole: true

RPi4 + UEFI + grub + kernel 5.12.6-1-MANJARO-ARM aarch64

[anatol]

Thanks for the report. Do you use the latest booster release?

I wonder if it is some sort of circular dependencies between kernel modules.

Could you please add booster.debug boot flag to your grub config, reboot the system and provide the log?

[anatol]

Also add booster.disable_concurrent_module_loading boot flag and let me know if it makes any difference.

[0n0w1c]

booster 0.5-1

Adding the debug, I can see there is a difference in output but the scroll is too fast and ended with the exact same information as the pic above. However, adding the second made a difference, it looks to be much more helpful. https://forum.manjaro.org/uploads/default/optimized/3X/1/c/1ccf5c5ce92d21fdaef6588b276d0cbff58e3346_2_281x500.jpeg

Maybe you can see it more clearly on the forum post: https://forum.manjaro.org/t/booster-for-initramfs/68596/7

[anatol]

Alright, the issue is missing sha2_ce module. It should be added to the image. A quick solution for it is to add it to modules config option.

I wonder what functionality requires this module and why it is not added to the image. Could you please show the output of

cd /usr/lib/modules/`uname -r`
grep sha2_ce -R

at the RPi system?

[0n0w1c]

[root@abbynormal 5.12.6-1-MANJARO-ARM]# grep sha2_ce -R
grep: kernel/arch/arm64/crypto/sha2-ce.ko: binary file matches
modules.alias:alias crypto-sha256 sha2_ce
modules.alias:alias sha256 sha2_ce
modules.alias:alias crypto-sha224 sha2_ce
modules.alias:alias sha224 sha2_ce
modules.alias:alias cpu:type:*:feature:*0006* sha2_ce
grep: modules.alias.bin: binary file matches

I'll try adding the module and report back.

The module was added, however the module does not load.

# modprobe sha2_ce
modprobe: ERROR: could not insert 'sha2_ce': No such device

[anatol]

Does the modprobe comes from booster log? Could you please post more of the booster logs?

[0n0w1c]

I do not think the root filesystem is loaded at this point. But where should the log be located?

I ran the modprobe from the command line, just to see if would load on a booted OS.

[anatol]

I mean post the output of booster init as a picture like you did before.

[0n0w1c]

Nothing has changed from this one: https://forum.manjaro.org/uploads/default/original/3X/1/c/1ccf5c5ce92d21fdaef6588b276d0cbff58e3346.jpeg

It loads the next module. so not sure this module is related to the issue.

It appears to be the loading of xxhash_generic, which does load with a dracut booted system but is not loaded on boot.

[anatol]

It means that sha2_ce still not in the booster image.

Set following config option modules: -*,hid_apple,kernel/fs/btrfs/btrfs.ko,kernel/drivers/block/zram/zram.ko,kernel/arch/arm64/crypto/sha2-ce.ko, then run booster -debug and post its output. Make sure that the output image contains sha2_ce module and then copy it to place where it is expected by grub.

[0n0w1c]

# lsinitcpio booster.img | grep sha2_ce
usr/lib/modules/sha2_ce.ko

It is in the booster image by default, no need to add it manually. But I did do that earlier, but it had no change.

Wait, sorry I mis-read. I will do as you ask.

[anatol]

Alright, so the sha2_ce module is in image and booster init tries to load it. But for some reason finit() fails with ENODEV kernel error. Huh... And dmesg does not have any additional information about this error. This is super-weird.

[0n0w1c]

# booster -debug -force
active host modules: [uio_pdrv_genirq thermal cpuidle cryptomgr sha256_arm64 xor syscopyarea dns_resolver spurious gpu_sched battery vt videodev rpivid_mem tcp_cubic bcm2835_codec xhci_hcd 8250 vc_sm_cma i2c_hid kgdboc soundcore srcutree sdhci rcupdate crypto_user random genet drm_kms_helper smsc95xx sysimgblt sysrq bcm2835_mmc overlay v3d i2c_bcm2835 processor scsi_mod videobuf2_vmalloc kernel backlight nvmem_rmem mousedev usbhid vc4 hid_apple uio gpiolib_acpi sch_fq_codel nfsv4 snd_bcm2835 nvme videobuf2_dma_contig acpi_power_meter bcm2835_mmal_vchiq blake2b_generic raid6_pq mmcblk efivarfs bcm2708_fb auth_rpcgss videobuf2_v4l2 keyboard zswap scsi_transport_iscsi crct10dif_ce f2fs binder snd_timer fb hid netpoll apparmor brd usbcore nfs_layout_flexfiles usb_storage blk_cgroup nvme_core pcie_aspm pnp rfkill joydev snd_compress cpufreq lockd ipv6 sunrpc btrfs cachefiles sha256_generic brcmfmac crc_t10dif fuse raspberrypi_hwmon bcm2835_v4l2 fb_sys_fops iscsi_ibft videobuf2_common configfs xxhash_generic snd_soc_core loop cec x_tables rcutree sysfillrect button lan78xx kvm mc brcmutil block snd_pcm xz_dec snd kdb fscache nfs_layout_nfsv41_files ip_tables module firmware_class binder_alloc acpi fscrypto rng_core printk v4l2_mem2mem cfg80211 i2c_dev watchdog vc_mem debug_core nfs workqueue dwc_otg xor_neon videobuf2_memops i2c_bcm2708 drm bcm2835_isp snd_pcm_dmaengine]
activate module nfs_layout_nfsv41_files
activate module configfs
activate module lockd
activate module nfsv4
activate module nfs_layout_flexfiles
activate module nfs
activate module btrfs
activate module fuse
activate module cachefiles
activate module efivarfs
activate module overlay
activate module fscache
activate module f2fs
activate module cryptomgr
activate module sha256_generic
activate module blake2b_generic
activate module xxhash_generic
activate module crypto_user
activate module xor
activate module genet
activate module dwc_otg
activate module xhci_hcd
activate module usbhid
deactivate module sha256_generic
deactivate module blake2b_generic
deactivate module xxhash_generic
deactivate module efivarfs
deactivate module crypto_user
deactivate module overlay
deactivate module fscache
deactivate module genet
deactivate module dwc_otg
deactivate module f2fs
deactivate module xhci_hcd
deactivate module nfs_layout_nfsv41_files
deactivate module configfs
deactivate module xor
deactivate module usbhid
deactivate module lockd
deactivate module nfsv4
deactivate module nfs_layout_flexfiles
deactivate module nfs
deactivate module btrfs
deactivate module fuse
deactivate module cachefiles
deactivate module cryptomgr
activate module hid_apple
activate module btrfs
activate module zram
activate module sha2_ce
softdep: unable to resolve module name aead2
softdep: unable to resolve module name aes
softdep: unable to resolve module name nls
softdep: unable to resolve module name crc32c
no matches found for a device alias 'of:NdmaT(null)Cbrcm,bcm2711-dma'
no matches found for a device alias 'of:Navs-monitorT(null)Cbrcm,bcm2711-avs-monitorCsysconCsimple-mfd'
no matches found for a device alias 'of:NdmaT(null)Cbrcm,bcm2835-dma'
no matches found for a device alias 'platform:bcm2835-camera'
no matches found for a device alias 'of:NmailboxT(null)Cbrcm,bcm2835-mbox'
no matches found for a device alias 'of:NwatchdogT(null)Cbrcm,bcm2835-pmCbrcm,bcm2835-pm-wdt'
no matches found for a device alias 'of:NgpioT(null)Cbrcm,bcm2711-gpio'
no matches found for a device alias 'of:Nfixedregulator_3v3T(null)Cregulator-fixed'
no matches found for a device alias 'platform:rtc-efi'
no matches found for a device alias 'usb:v174Cp55AAd0100dc00dsc00dp00ic08isc06ip62in00'
no matches found for a device alias 'of:Nsd_vcc_regT(null)Cregulator-fixed'
no matches found for a device alias 'platform:bcm2835-codec'
no matches found for a device alias 'platform:bcm2835-power'
no matches found for a device alias 'of:NcprmanT(null)Cbrcm,bcm2711-cprman'
no matches found for a device alias 'of:NauxT(null)Cbrcm,bcm2835-aux'
no matches found for a device alias 'of:NchosenT(null)Csimple_bus'
no matches found for a device alias 'of:Nlocal_intcT(null)Cbrcm,bcm2836-l1-intc'
no matches found for a device alias 'sdio:c00v02D0dA9A6'
no matches found for a device alias 'of:NresetT(null)Craspberrypi,firmware-reset'
no matches found for a device alias 'of:NtimerT(null)Carm,armv8-timer'
no matches found for a device alias 'of:Nv3dbusT(null)Csimple-bus'
no matches found for a device alias 'usb:v1D6Bp0002d0512dc09dsc00dp01ic09isc00ip00in00'
no matches found for a device alias 'of:NscbT(null)Csimple-bus'
no matches found for a device alias 'of:NpsciT(null)Carm,psci-1.0'
no matches found for a device alias 'of:Nfixedregulator_5v0T(null)Cregulator-fixed'
no matches found for a device alias 'of:NledsT(null)Cgpio-leds'
no matches found for a device alias 'usb:v045Ep00CBd0100dc00dsc00dp00ic03isc01ip02in00'
no matches found for a device alias 'of:NethernetT(null)Cbrcm,bcm2711-genet-v5'
no matches found for a device alias 'of:Nh264-decoderT(null)Craspberrypi,rpivid-h264-decoder'
no matches found for a device alias 'of:Nbcm2835_audioT(null)Cbrcm,bcm2835-audio'
no matches found for a device alias 'of:NmailboxT(null)Cbrcm,bcm2711-vchiq'
no matches found for a device alias 'platform:efi-framebuffer'
no matches found for a device alias 'of:NsocT(null)Csimple-bus'
no matches found for a device alias 'of:NpowerT(null)Craspberrypi,bcm2835-power'
no matches found for a device alias 'of:Ni2cT(null)Cbrcm,bcm2711-i2cCbrcm,bcm2835-i2c'
no matches found for a device alias 'pci:v00001106d00003483sv00001106sd00003483bc0Csc03i30'
no matches found for a device alias 'input:b0003v05ACp024Fe0111-e0,1,4,11,14,k71,72,73,74,75,77,78,79,7A,7B,7C,7D,7E,7F,80,81,82,83,84,85,86,87,88,89,8A,8C,8E,96,98,9E,9F,A1,A3,A4,A5,A6,AD,B0,B1,B2,B3,B4,B7,B8,B9,BA,BB,BC,BD,BE,BF,C0,C1,C2,CC,E0,E1,E3,E4,E5,E6,F0,1D0,ram4,l0,1,2,3,4,sfw'
no matches found for a device alias 'input:b0003v05ACp024Fe0111-e0,1,2,3,4,14,k71,72,73,74,75,77,78,7F,80,82,83,84,85,86,87,88,89,8A,8B,8C,8E,8F,90,96,98,9B,9C,9E,9F,A1,A3,A4,A5,A6,A7,A8,A9,AB,AC,AD,AE,B0,B1,B2,B5,B6,B7,B8,B9,BA,BB,BC,BD,BE,BF,C0,C1,C2,CE,CF,D0,D1,D2,D4,D8,D9,DB,DF,E0,E1,E4,E5,E6,E7,E8,E9,EA,EB,F0,F1,F4,100,110,111,112,161,162,166,16A,16E,172,174,176,177,178,179,17A,17B,17C,17D,17F,180,182,183,185,188,189,18C,18D,18E,18F,190,191,192,193,195,197,198,199,19A,19C,1A0,1A1,1A2,1A3,1A4,1A5,1A6,1A7,1A8,1A9,1AA,1AB,1AC,1AD,1AE,1AF,1B0,1B1,1B7,1BA,240,241,242,243,244,245,246,247,248,250,251,260,261,262,263,264,265,r0,1,6,8,B,C,a20,28,m4,lsfw'
no matches found for a device alias 'hid:b0003g0001v0000045Ep000000CB'
no matches found for a device alias 'scsi:t-0x00'
no matches found for a device alias 'of:NmmcnrT(null)Cbrcm,bcm2835-mmcCbrcm,bcm2835-sdhci'
no matches found for a device alias 'platform:raspberrypi-cpufreq'
no matches found for a device alias 'of:Nemmc2busT(null)Csimple-bus'
no matches found for a device alias 'dmi:bvnhttps//github.com/pftf/RPi4:bvrUEFIFirmwarev1.26:bd04/16/2021:br1.26:efr21.4:svnRaspberryPiFoundation:pnRaspberryPi4ModelB:pvrD03114:rvnSonyUK:rnRaspberryPi4ModelB:rvrD03114:cvnSonyUK:ct34:cvrRaspberryPi4ModelB:'
no matches found for a device alias 'usb:v1D6Bp0003d0512dc09dsc00dp03ic09isc00ip00in00'
no matches found for a device alias 'usb:v0BDAp0411d0127dc09dsc00dp03ic09isc00ip00in00'
no matches found for a device alias 'of:NthermalT(null)Cbrcm,bcm2711-thermal'
no matches found for a device alias 'of:NrngT(null)Cbrcm,bcm2711-rng200'
no matches found for a device alias 'usb:v0BDAp5411d0127dc09dsc00dp02ic09isc00ip02in00'
no matches found for a device alias 'of:NgpuT(null)Cbrcm,bcm2711-vc5'
no matches found for a device alias 'of:NpcieTpciCbrcm,bcm2711-pcie'
no matches found for a device alias 'input:b0003v045Ep00CBe0111-e0,1,2,4,k110,111,112,r0,1,8,B,am4,lsfw'
no matches found for a device alias 'platform:bcm2835-isp'
no matches found for a device alias 'platform:efivars'
no matches found for a device alias 'platform:vcsm-cma'
no matches found for a device alias 'of:NclocksT(null)Craspberrypi,firmware-clocks'
no matches found for a device alias 'of:Nv3dT(null)Cbrcm,2711-v3d'
no matches found for a device alias 'pci:v000014E4d00002711sv00000000sd00000000bc06sc04i00'
no matches found for a device alias 'of:Nsd_io_1v8_regT(null)Cregulator-gpio'
no matches found for a device alias 'platform:snd-soc-dummy'
no matches found for a device alias 'platform:bcm2835-wdt'
no matches found for a device alias 'of:NfirmwareT(null)Craspberrypi,bcm2835-firmwareCsimple-mfd'
no matches found for a device alias 'of:NvcsmT(null)Craspberrypi,bcm2835-vcsm'
no matches found for a device alias 'of:Nrpivid-local-intcT(null)Craspberrypi,rpivid-local-intc'
no matches found for a device alias 'usb:v05ACp024Fd0102dc00dsc00dp00ic03isc01ip02in01'
no matches found for a device alias 'of:NtimerT(null)Cbrcm,bcm2835-system-timer'
no matches found for a device alias 'of:NfirmwarekmsT(null)Craspberrypi,rpi-firmware-kms-2711'
no matches found for a device alias 'of:NgpioT(null)Craspberrypi,firmware-gpio'
no matches found for a device alias 'platform:reg-dummy'
no matches found for a device alias 'of:Nemmc2T(null)Cbrcm,bcm2711-emmc2'
no matches found for a device alias 'of:NmdioT(null)Cbrcm,genet-mdio-v5'
no matches found for a device alias 'sdio:c02v02D0dA9A6'
no matches found for a device alias 'platform:raspberrypi-hwmon'
no matches found for a device alias 'platform:Fixed MDIO bus'
no matches found for a device alias 'platform:kgdboc'
no matches found for a device alias 'of:NphyT(null)Cusb-nop-xceiv'
no matches found for a device alias 'of:Nvp9-decoderT(null)Craspberrypi,rpivid-vp9-decoder'
no matches found for a device alias 'cpu:type:aarch64:feature:,0000,0001,0002,0007,000B'
no matches found for a device alias 'of:NnvramT(null)Craspberrypi,bootloader-configCnvmem-rmem'
no matches found for a device alias 'platform:cpufreq-dt'
no matches found for a device alias 'of:NgpiomemT(null)Cbrcm,bcm2835-gpiomem'
no matches found for a device alias 'of:Narm-pmuT(null)Carm,cortex-a72-pmuCarm,armv8-pmuv3'
no matches found for a device alias 'usb:v05ACp024Fd0102dc00dsc00dp00ic03isc01ip01in00'
no matches found for a device alias 'usb:v2109p3431d0421dc09dsc00dp01ic09isc00ip00in00'
no matches found for a device alias 'of:Nhevc-decoderT(null)Craspberrypi,rpivid-hevc-decoder'
no matches found for a device alias 'platform:regulatory'
module 'mmcblk' currently used at the host but was not added to the image
module 'bcm2708_fb' currently used at the host but was not added to the image
module 'blk_cgroup' currently used at the host but was not added to the image
module 'v3d' currently used at the host but was not added to the image
module 'bcm2835_v4l2' currently used at the host but was not added to the image
module 'v4l2_mem2mem' currently used at the host but was not added to the image
module 'overlay' currently used at the host but was not added to the image
module 'watchdog' currently used at the host but was not added to the image
module 'vc_mem' currently used at the host but was not added to the image
module 'apparmor' currently used at the host but was not added to the image
module 'crypto_user' currently used at the host but was not added to the image
module 'mousedev' currently used at the host but was not added to the image
module 'rng_core' currently used at the host but was not added to the image
module 'tcp_cubic' currently used at the host but was not added to the image
module 'brcmutil' currently used at the host but was not added to the image
module 'processor' currently used at the host but was not added to the image
module 'kernel' currently used at the host but was not added to the image
module 'brd' currently used at the host but was not added to the image
module 'lockd' currently used at the host but was not added to the image
module 'fb_sys_fops' currently used at the host but was not added to the image
module 'gpu_sched' currently used at the host but was not added to the image
module 'random' currently used at the host but was not added to the image
module 'bcm2835_mmc' currently used at the host but was not added to the image
module 'snd_pcm_dmaengine' currently used at the host but was not added to the image
module 'rpivid_mem' currently used at the host but was not added to the image
module 'efivarfs' currently used at the host but was not added to the image
module 'rcutree' currently used at the host but was not added to the image
module '8250' currently used at the host but was not added to the image
module 'dns_resolver' currently used at the host but was not added to the image
module 'crct10dif_ce' currently used at the host but was not added to the image
module 'fuse' currently used at the host but was not added to the image
module 'iscsi_ibft' currently used at the host but was not added to the image
module 'thermal' currently used at the host but was not added to the image
module 'usbcore' currently used at the host but was not added to the image
module 'x_tables' currently used at the host but was not added to the image
module 'lan78xx' currently used at the host but was not added to the image
module 'snd_pcm' currently used at the host but was not added to the image
module 'srcutree' currently used at the host but was not added to the image
module 'i2c_bcm2835' currently used at the host but was not added to the image
module 'usbhid' currently used at the host but was not added to the image
module 'f2fs' currently used at the host but was not added to the image
module 'pnp' currently used at the host but was not added to the image
module 'battery' currently used at the host but was not added to the image
module 'genet' currently used at the host but was not added to the image
module 'videobuf2_vmalloc' currently used at the host but was not added to the image
module 'loop' currently used at the host but was not added to the image
module 'i2c_hid' currently used at the host but was not added to the image
module 'cpufreq' currently used at the host but was not added to the image
module 'brcmfmac' currently used at the host but was not added to the image
module 'usb_storage' currently used at the host but was not added to the image
module 'smsc95xx' currently used at the host but was not added to the image
module 'sch_fq_codel' currently used at the host but was not added to the image
module 'nfsv4' currently used at the host but was not added to the image
module 'rfkill' currently used at the host but was not added to the image
module 'nfs_layout_nfsv41_files' currently used at the host but was not added to the image
module 'vc_sm_cma' currently used at the host but was not added to the image
module 'gpiolib_acpi' currently used at the host but was not added to the image
module 'ipv6' currently used at the host but was not added to the image
module 'mc' currently used at the host but was not added to the image
module 'fscache' currently used at the host but was not added to the image
module 'sdhci' currently used at the host but was not added to the image
module 'i2c_bcm2708' currently used at the host but was not added to the image
module 'nfs' currently used at the host but was not added to the image
module 'auth_rpcgss' currently used at the host but was not added to the image
module 'keyboard' currently used at the host but was not added to the image
module 'scsi_transport_iscsi' currently used at the host but was not added to the image
module 'vt' currently used at the host but was not added to the image
module 'videodev' currently used at the host but was not added to the image
module 'soundcore' currently used at the host but was not added to the image
module 'binder' currently used at the host but was not added to the image
module 'nvme_core' currently used at the host but was not added to the image
module 'sunrpc' currently used at the host but was not added to the image
module 'cpuidle' currently used at the host but was not added to the image
module 'block' currently used at the host but was not added to the image
module 'videobuf2_dma_contig' currently used at the host but was not added to the image
module 'scsi_mod' currently used at the host but was not added to the image
module 'bcm2835_mmal_vchiq' currently used at the host but was not added to the image
module 'crc_t10dif' currently used at the host but was not added to the image
module 'drm' currently used at the host but was not added to the image
module 'uio_pdrv_genirq' currently used at the host but was not added to the image
module 'uio' currently used at the host but was not added to the image
module 'cachefiles' currently used at the host but was not added to the image
module 'binder_alloc' currently used at the host but was not added to the image
module 'printk' currently used at the host but was not added to the image
module 'cfg80211' currently used at the host but was not added to the image
module 'bcm2835_isp' currently used at the host but was not added to the image
module 'vc4' currently used at the host but was not added to the image
module 'sysrq' currently used at the host but was not added to the image
module 'acpi_power_meter' currently used at the host but was not added to the image
module 'kvm' currently used at the host but was not added to the image
module 'xhci_hcd' currently used at the host but was not added to the image
module 'rcupdate' currently used at the host but was not added to the image
module 'fb' currently used at the host but was not added to the image
module 'nfs_layout_flexfiles' currently used at the host but was not added to the image
module 'videobuf2_common' currently used at the host but was not added to the image
module 'acpi' currently used at the host but was not added to the image
module 'kdb' currently used at the host but was not added to the image
module 'snd_bcm2835' currently used at the host but was not added to the image
module 'snd_timer' currently used at the host but was not added to the image
module 'sha256_generic' currently used at the host but was not added to the image
module 'fscrypto' currently used at the host but was not added to the image
module 'drm_kms_helper' currently used at the host but was not added to the image
module 'nvme' currently used at the host but was not added to the image
module 'configfs' currently used at the host but was not added to the image
module 'snd_soc_core' currently used at the host but was not added to the image
module 'ip_tables' currently used at the host but was not added to the image
module 'workqueue' currently used at the host but was not added to the image
module 'bcm2835_codec' currently used at the host but was not added to the image
module 'sysimgblt' currently used at the host but was not added to the image
module 'xz_dec' currently used at the host but was not added to the image
module 'firmware_class' currently used at the host but was not added to the image
module 'pcie_aspm' currently used at the host but was not added to the image
module 'nvmem_rmem' currently used at the host but was not added to the image
module 'zswap' currently used at the host but was not added to the image
module 'hid' currently used at the host but was not added to the image
module 'netpoll' currently used at the host but was not added to the image
module 'joydev' currently used at the host but was not added to the image
module 'i2c_dev' currently used at the host but was not added to the image
module 'debug_core' currently used at the host but was not added to the image
module 'backlight' currently used at the host but was not added to the image
module 'sysfillrect' currently used at the host but was not added to the image
module 'kgdboc' currently used at the host but was not added to the image
module 'spurious' currently used at the host but was not added to the image
module 'videobuf2_v4l2' currently used at the host but was not added to the image
module 'snd_compress' currently used at the host but was not added to the image
module 'raspberrypi_hwmon' currently used at the host but was not added to the image
module 'cec' currently used at the host but was not added to the image
module 'module' currently used at the host but was not added to the image
module 'cryptomgr' currently used at the host but was not added to the image
module 'button' currently used at the host but was not added to the image
module 'snd' currently used at the host but was not added to the image
module 'dwc_otg' currently used at the host but was not added to the image
module 'videobuf2_memops' currently used at the host but was not added to the image
module 'syscopyarea' currently used at the host but was not added to the image
enabling virtual console
detected language - 'en_US.UTF-8'
/etc/vconsole.conf does not provide FONT settings, skip vconsole font configuration

[0n0w1c]

# dmesg | grep sha2

# dmesg | grep -i error

With dracut the boot is clean.

[anatol]

I see that your system uses sha256_generic but it was added to the booster image. Could you please try adding it and force its loading using following config option: modules_force_load: sha256_generic?

[0n0w1c]

I have tried to exclude xxhash_generic with modules: -*,hid_apple,kernel/fs/btrfs/btrfs.ko,kernel/drivers/block/zram/zram.ko,-kernel/crypto/xxhash_generic.ko

However, it continues to be included in the image. And neither sha2_ce nor xxhash_generic are mentioned in the booster debug output but they are included in the image.

[0n0w1c]

# lsmod | grep sha
sha256_generic         16384  0
sha256_arm64           28672  0

The module is loaded but it appears nothing is using it, or at least that is my interpretation of the 0.

And yes, I will now add sha256_genreric.

[anatol]

Does it mean that with added sha256_genreric module you are able to boot booster at RPi4?

[0n0w1c]

# lsmod | grep sha
sha256_generic         16384  0
sha256_arm64           28672  0

The module is loaded but it appears nothing is using it, or at least that is my interpretation of the 0.

And yes, I will now add sha256_genreric.

[0n0w1c]

No, I have not been able to successfully boot with booster. When I boot to post here, I am using a dracut initramfs.

Is there a way to prevent a module from loading? I would like to try booting without the xxhash_generic. However, it will need to wait until the morning. I must call it a night.

[anatol]

That xxhash_generic is probably loaded as a dependency of some other module (like btrfs).

There is no out-of-box way to disable a specific module loading. But if you really want it - modify init/module.go method loadModuleUnlocked() and right after for _, module := range modules { add if module == "module.to.ignore" {continue}.

I wish I had an access to a RaspberryPi4 so I can debug the issue myself. What I would do in this case is reduced booster config a bit:

1. disabled network. booster needs network only if the filesystem is remote or unlocker requires a network service like Tang
2. removed modules: -* and started with list of modules selected by booster
3. use universal images universal: true it will create much larger image but it also will contain a lot more different modules.
4. removed modules_force_load

Then I would add extra_files: busybox - this enables emergency shell.

Also what is GRUB insmod part_gpt; insmod btrfs config options? Are they really needed taking into account that initramfs loads btrfs modules anyway?

[0n0w1c]

The reason for the grub configs? Sort of self-documenting what my booting requires of grub.

Yes, btrfs does bring in xxhash_generic, I can't do much about that.

I tried the 4 suggestions above, but no change. As I can not load the sha2_ce module, I expect the RPi4 does not support the crypto extension. I have asked for it to be removed from the next rpi rc kernel build and I will try again with that kernel. Not sure when that will be, hopefully in the next several days.

[anatol]

I wonder why booster loads it at all. There must be some module requires sha2_ce or one of its aliases (sha256, sha224,..).

Where I can find image for the RPi4 you use? I wan to unpack it and see the kernel modules info it has.

[anatol]

Yes, btrfs does bring in xxhash_generic, I can't do much about that.

Per logs here https://github.com/anatol/booster/issues/87#issuecomment-852655910 you have xxhash_generic in the list of the loaded modules. From the other side sha2_ce module is not used at your dracut system but booster pulls it anyway.

[0n0w1c]

https://github.com/manjaro-arm/rpi4-images/releases/download/21.04/Manjaro-ARM-kde-plasma-rpi4-21.04.img.xz That will get you close, but I am currently running the linux-rpi4-mainline kernel package.

git clone https://gitlab.manjaro.org/manjaro-arm/packages/core/linux-rpi4-mainline.git This will get you the kernel config for the linux-rpi4-mainline

And per #87, yes. At one time... I have tried several configs, trying to get a different result.

[0n0w1c]

The new screenshot, with the sha2_ce module removed from the kernel build... now it hangs at this point: https://forum.manjaro.org/uploads/default/original/3X/0/4/040234c55b5da6297b0dcb427b5053f66d32c18d.jpeg

However I was able to Ctrl-Alt-Delete and it rebooted... so better. Hopefully the new screenshot gives you an idea of something else for me to try.

[0n0w1c]

Eh, I see the compression did not help the image any... sorry. It hangs just after loading the hid_apple module which is for my keyboard but I can change to a logitech and remove the hid_apple module from the config. Speaking of, here is the config used for this booster.img:

$ cat /etc/booster.yaml
universal: true
modules: hid_apple,kernel/fs/btrfs/btrfs.ko,kernel/drivers/block/zram/zram.ko,kernel/crypto/sha256_generic.ko
compression: zstd
strip: false
extra_files: mount,fsck,fsck.btrfs,btrfsck,btrfs,zramctl
vconsole: true

And my grub config:

$ cat /boot/grub/grub.cfg (clipped)
menuentry "Manjaro Booster Test" {
          echo "Manjaro Booster Test booting..."
          insmod part_gpt
          insmod btrfs
          linux (hd0,gpt3)/@/boot/kernel8.img root=UUID=da0a03c1-016b-4782-824e-fe85ac1642c7 rw rootflags=defaults,ssd,compress-force=zstd,subvol=@ rootwait snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=0 console=serial0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=serial0,115200 audit=0 booster.debug booster.disable_concurrent_module_loading modprobe.blacklist=xxhash_generic
          initrd (hd0,gpt3)/@/boot/booster.img
          boot
}

[0n0w1c]

I removed everything to the right of audit=0 and then rebooted. Now I see what might be the next issue:

[    2.632892] Btrfs loaded, crc32c=crc32c-generic, zoned=yes
fsck for /dev/sda3 failed with code 0x8
[    2.650836] booster: fsck for /dev/sda3 failed with code 0x8

Which is followed by the loading of the hid_apple module and the subsequent hang.

Edit: I removed the hid_apple and rebuilt booster.img. Now the last console message are the failed fsck. Oddly, minus the booster: message echoing the failure. Probably just a buffering thing.

$ cat /etc/booster.yaml
universal: true
modules: kernel/fs/btrfs/btrfs.ko,kernel/drivers/block/zram/zram.ko
compression: zstd
strip: false
extra_files: mount,fsck,fsck.btrfs,btrfsck,btrfs,zramctl
vconsole: true

[anatol]

The problem you see (system hangs) is because fsck fails with error 8 which is quoting the manpage Operational error, eg. device does not exist Huh.. What are the logs right before the btrfs part? Do you see anything related to discovering block device sda or sda3?

Also add busybox to extra_files and set mount_timeout to some small value e.g. 40s. Once the timeout fires it enters the emergency shell where you have more opportunities to inspect the system e.g. check dmesg from the beginning of the boot process.

message echoing the failure

The messages are printed to stdout and to dmesg (and in verbose mode end up at stdout as well).

[0n0w1c]

Yes, it discovers /dev/sda[1-5] /dev/sda attached scsi disk

I'll do as you suggest and report back.

[0n0w1c]

/dev/sda
/dev/sda1
...
/dev/sda5

All exist in /dev. I performed btrfs check /dev/sda3 and it successfully passed, no errors.

/usr/bin/fsck.btrfs does not run, /bin/sh does not exist, as specified in the script.

How does booster perform the fsck on btrfs?

[anatol]

booster calls fsck $DEVICE.

Given the information above it looks like:

1. fsck.btrfs is no-op (surprisingly for me)
2. fsck.btrfs needs /bin/sh dependency.

For now just remove fsck binaries from the image and try again.

In the future we need some other way to run fsck automatically. I wonder what dracut uses for fsck-ing.

[0n0w1c]

Maybe consult fstab to see a fsck is enabled? Which is not in the image. A proper btrfs fsck takes a while, most do not check it on boot. fsck.btrfs is a do nothing script.

$ /usr/bin/fsck.btrfs
If you wish to check the consistency of a BTRFS filesystem or
repair a damaged filesystem, see btrfs(8) subcommand 'check'.

[0n0w1c]

Without the fsck binaries:

BTRFS: device label MNJRO_ROOT devid 1 transid 85243 /dev/sda3 scanned by init (1)
BTRFS error (device sda3) unrecognized mount option 'defaults'
BTRFS error (device sda3): open_ctree failed
mount(/dev/sda3): invalid argument
Timeout waiting for root filesystem
booster: mount(/dev/sda3): invalid argument

[0n0w1c]

I removed defaults from rootflags and now I can boot successfully with booster.

[anatol]

You need to replace defaults mount option with its explicit mount flags equivalent: rw,suid,dev,exec,auto,nouser,async

[anatol]

I removed defaults from rootflags and now I can boot successfully with booster.

Woot, congrats! Now I need to unwind this discussion thread and file tickets for the issues been found.

[0n0w1c]

Thank you for all of your assistance. I would like to try booster with tftp + nfsroot + systemd-overlayfs, should that be supported? It is not implemented with mkinitcpio, and why I use dracut.

[0n0w1c]

rw,suid,dev,exec,auto,nouser,async

I changed two things so not sure, I switched to universal: false and I added the above to my grub.cfg

But auto and nouser generated the same errors as defaults, so I removed them and all is good again.

[0n0w1c]

I also would like to add tiny sshd, to be able to remotely unlock LUKS disks. Would this possible?

[anatol]

I would like to try booster with tftp + nfsroot + systemd-overlayfs, should that be supported?

I doubt that it will work out of box.

1. Current booster architecture expects that each filesystem has an underlying block device. Which is not true for NFS. We need to extend booster's udev events processing and handle "network is up" event as "system is ready for network filesystems (NFS/SSHFS/...). Could you please file a ticket for NFS support? I am not an NFS user so detailed steps how to setup tftp/nfs would be super handy for me.
2. I've actually never heard of systemd-overlayfs. Is it some fork of overlayfs? What role systemd plays here?

[anatol]

I also would like to add tiny sshd, to be able to remotely unlock LUKS disks. Would this possible?

Currently booster does not support such use-case. Adding ssh server to the image brings a lot of complexity. Instead I plan to implement "reverse-Tang" feature that relies on much simpler key derivation protocol. See https://github.com/anatol/booster/issues/24

BTW booster supports network binding feature using Tang protocol. In this case booster boots machine and contacts Tang server to derive the unlocking key. Tang server runs in the local network and contains private unlock master key.

i.e. in this case of sshd boot stops and then administrator connects machine to unlock it. In case of tang the machine contacts Tang server to derive the unlocking key. It is not exactly the same as you are looking for but it might work for you in the meanwhile until reverse-tang feature is implemented.

[0n0w1c]

kernel command line parameter: systemd.volatile=overlay https://www.freedesktop.org/software/systemd/man/systemd-fstab-generator.html#systemd.volatile= Very simple to use and works great. Better than the other overlayfs configs I have seen, at least for my purposes.

When you run df, this is all that is visible for the rootfs.

overlay       58070016 33716764  23024948  60% /

No upper/lower mounts, very clean.

I will have to read up on the reverse-Tang, I am not familiar with it.

Sure, I'll be happy to open a ticket for NFS.

[anatol]

I will have to read up on the reverse-Tang

Note that "tang" is what supported now. "reverse-tang" is a term that I came up. "reverse-tang" is just like a regular Tang protocol but connection initiated by server (master key holder).

[0n0w1c]

After a little reading and thinking about your reverse-tang plan, it is a fine idea. +1

[anatol]

@0n0w1c I believe your original issue has been fixed and this issue can be closed now. The other requests you had )overlay and remove unlock support) are tracked via its own tickets.