rear / rear

Relax-and-Recover - Linux bare metal disaster recovery and system migration solution (cfr. mksysb, ignite)
http://relax-and-recover.org/
GNU General Public License v3.0
921 stars 250 forks source link

OUTPUT=PORTABLE on SLES15/SP5 in `rear recover` dracut fails to build initramfs #3263

Open saschalucas opened 2 months ago

saschalucas commented 2 months ago

BACKUP=EXTERNAL EXTERNAL_RESTORE=/root/commvault-restore.sh

FIRMWARE_FILES=( 'no' ) MODULES=( 'no_modules' )

DISKS_TO_BE_WIPED='/dev/vd[a-z] /dev/sd[a-z]' USER_INPUT_TIMEOUT=30


* Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): `VM (esx)`

* System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): `x86_64`

* Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): `BIOS/grub`

* Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): `local disk`

* Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):
```text
NAME                        KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL  SIZE MOUNTPOINT
/dev/sda                    /dev/sda                 disk                     20G 
|-/dev/sda1                 /dev/sda1 /dev/sda       part                      8M 
`-/dev/sda2                 /dev/sda2 /dev/sda       part LVM2_member         20G 
  |-/dev/mapper/system-swap /dev/dm-0 /dev/sda2      lvm  swap                 4G [SWAP]
  `-/dev/mapper/system-root /dev/dm-1 /dev/sda2      lvm  ext4                19G /
/dev/sdb                    /dev/sdb                 disk LVM2_member         10G 
|-/dev/mapper/system-swap   /dev/dm-0 /dev/sdb       lvm  swap                 4G [SWAP]
`-/dev/mapper/system-data   /dev/dm-2 /dev/sdb       lvm  ext4                 7G /data
/dev/sr0                    /dev/sr0            ata  rom                    1024M 

2024-07-02 13:11:06.639258185 Including finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh



* Workaround, if any: `-`

* Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files): 
[rear-mkrescue-server1005.log](https://github.com/user-attachments/files/16067384/rear-mkrescue-server1005.log)
[rear-recover-server1005.log](https://github.com/user-attachments/files/16067386/rear-recover-server1005.log)
logfiles of mkrescue and recover attached, however without -D.
gdha commented 2 months ago

@saschalucas Did you check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed?

saschalucas commented 2 months ago

@saschalucas Did you check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed?

yes. /rear/var/log/rear/rear-server1005-rescue.log is attached as rear-recover-server1005.log. Sorry for the renaming ... but I found rescue didn't match the rear workflow. In the log itself there is nothing else like on screen and like mentioned above in the issue:

2024-07-02 13:11:01.185390577 Including finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh
2024-07-02 13:11:06.369117289 Recreating initrd with /usr/bin/dracut...
2024-07-02 13:11:06.632109467 Warning:
                              Failed to recreate initrd with /usr/bin/dracut.
                              Check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed
                              and decide if the recreated system will boot
                              with the initrd 'as is' from the backup restore.

2024-07-02 13:11:06.639258185 Including finalize/Linux-i386/610_EFISTUB_run_efibootmgr.sh

So the log contains no further hint what went wrong.

jsmeix commented 2 months ago

@saschalucas you need at least debug mode '-d' preferably full debugscript mode '-D' to be able to see what went wrong.

In general: Caution with possible secrets in a debug log file: When 'rear' is run via '-D' in debugscript mode it logs executed commands via the bash command 'set -x' that print commands and their arguments as they are executed. So in particular when arguments contain secret values (e.g. something like a password or whatever else) such secret values may appear in the log file. Also secrets may be stored in some other files like /var/lib/rear/layout/disklayout.conf or /var/lib/rear/layout/diskrestore.sh cf. [password=<password>] in the section "Disk layout file syntax" in doc/user-guide/06-layout-configuration.adoc online at https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc So before you attach your debug log file or other files here (GitHub is a public accessible place), inspect your files and verify that they do not accidentally cointain secrets.

jsmeix commented 2 months ago

@saschalucas because you use OUTPUT=PORTABLE: What rescue system do you use? I ask because

In portable mode ReaR assumes that all required software
is available in the rescue system. There are no further
checks for missing software. If you miss some software,
the recovery process will fail.

cf. https://github.com/rear/rear/blob/master/doc/user-guide/17-Portable-Mode.adoc

saschalucas commented 2 months ago

-D reveals the problem:

++ LogPrint 'Recreating initrd with /usr/bin/dracut...'
2024-07-03 18:29:35.377244440 Recreating initrd with /usr/bin/dracut...
++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force'

dracut: Cannot find module directory /lib/modules/6.6.14-1-lts/
dracut: and --no-kernel was not specified
++ LogPrintError 'Warning:
Failed to recreate initrd with /usr/bin/dracut.
Check '\''/rear/var/log/rear/rear-server1005-rescue.log'\'' why /usr/bin/dracut failed
and decide if the recreated system will boot
with the initrd '\''as is'\'' from the backup restore.
'
2024-07-03 18:29:35.760132926 Warning:
                              Failed to recreate initrd with /usr/bin/dracut.
                              Check '/rear/var/log/rear/rear-server1005-rescue.log' why /usr/bin/dracut failed
                              and decide if the recreated system will boot
                              with the initrd 'as is' from the backup restore.

The SLES version of finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh fails here, because it is looking for a kernel from the rescue system systemrescue-11.0. So I'm bitten by a side effect of the portable mode here.

For curiosity on Oracle Linux 8 dracut seems to be called in an other way. Under the same rescue system systemrescue-11.0 the recreation of initramfs seems to work:

2024-06-26 08:21:28.990404381 Including finalize/Fedora/550_rebuild_initramfs.sh
2024-06-26 08:21:28.994816038 Original OLD_INITRD_MODULES=(  )
2024-06-26 08:21:29.006200942 New INITRD_MODULES=' '
2024-06-26 08:21:29.029542327 Running dracut...
2024-06-26 08:21:44.491985680 Updated initrd with new drivers for kernel 4.18.0-513.18.1.0.1.el8_9.x86_64.
2024-06-26 08:21:44.500304291 Running dracut...
2024-06-26 08:21:59.546876691 Updated initrd with new drivers for kernel 4.18.0-513.9.1.el8_9.x86_64.
2024-06-26 08:21:59.556064290 Running dracut...
2024-06-26 08:22:15.147509223 Updated initrd with new drivers for kernel 5.15.0-201.135.6.el8uek.x86_64.
2024-06-26 08:22:15.156579498 Running dracut...
2024-06-26 08:22:30.757426667 Updated initrd with new drivers for kernel 5.15.0-205.149.5.1.el8uek.x86_64.
schlomo commented 1 month ago

@saschalucas the Fedora version iterates over all the existing initrd images:

https://github.com/rear/rear/blob/87d0b7f190ed758ef06f5d93e860581d89cc75b3/usr/share/rear/finalize/Fedora/550_rebuild_initramfs.sh#L70-L87

The SUSE version however doesn't:

https://github.com/rear/rear/blob/87d0b7f190ed758ef06f5d93e860581d89cc75b3/usr/share/rear/finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh#L52-L60

It calls dracut without any parameters which makes it use the current kernel version.

If you want I can rework the SUSE version of this script to be more like the Fedora version and not make assumptions about the running kernel version.

saschalucas commented 1 month ago

If you want I can rework the SUSE version of this script to be more like the Fedora version and not make assumptions about the running kernel version.

Yes please, that would be desirable to better support the new portable mode.