Handle default kernel cmdline if multiple boot entries for the default kernel are defined

matejmatuska commented 2 weeks ago

Instead of erroring out when grubby lists multiple entries for the default kernel, always use the args= and root= from the first one and create a post-upgrade report. The report instruct user to ensure those are the correct ones or to correct them. This can happen, for example, if MAKEDEBUG=yes is set in /etc/sysconfing/kernel.

Also the missing leapp.reporting.Report class is added to kernelcmdlineconfig actor produces tuple.

Jira: RHEL-46911

github-actions[bot] commented 2 weeks ago

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable. If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. However, here are additional useful commands for packit:

/packit test to re-run manually the default tests
/packit retest-failed to re-run failed tests manually
/packit test oamg/leapp#42 to run tests with leapp builds for the leapp PR#42 (default is latest upstream - main - build)

Note that first time contributors cannot run tests automatically - they need to be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones. To launch on-demand tests with packit:

/packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
/packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

pirat89 commented 2 weeks ago

/packit copr-build

pirat89 commented 1 week ago

Tested manually, kind of it works but I found that our solution is not good for such a setup - affecting all major IPU paths.

Reproducer:

# echo "MAKEDEBUG=yes" >> /etc/sysconfig/kernel
# grub2-mkconfig -o /boot/grub2/grub.cfg
# leapp upgrade --debug --reboot

Result

----------------------------------------------------------------------
Actor: kernelcmdlineconfig
Executed: True
Phase: Finalization
Started: 2024-10-16T10:19:10.629165Z
Produced messages:
    - Report
Executed shell commands:
    - ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64', '--args', 'net.ifnames=0', '--remove-args', 'enforcing=0']
    - ['grubby', '--info', '/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64']
Logs:
    2024-10-16 12:19:10.689 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: External command has started: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64', '--args', 'net.ifnames=0', '--remove-args', 'enforcing=0']
    2024-10-16 12:19:10.833 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: External command has finished: ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64', '--args', 'net.ifnames=0', '--remove-args', 'enforcing=0']
    2024-10-16 12:19:10.834 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: External command has started: ['grubby', '--info', '/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64']
    2024-10-16 12:19:10.943 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: index=0
    2024-10-16 12:19:10.945 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: kernel="/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64"
    2024-10-16 12:19:10.946 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: args="ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet $tuned_params net.ifnames=0"
    2024-10-16 12:19:10.947 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: root="/dev/mapper/rhel-root"
    2024-10-16 12:19:10.948 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: initrd="/boot/initramfs-4.18.0-553.22.1.el8_10.x86_64.img $tuned_initrd"
    2024-10-16 12:19:10.949 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: title="Red Hat Enterprise Linux (4.18.0-553.22.1.el8_10.x86_64) 8.10 (Ootpa)"
    2024-10-16 12:19:10.950 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: id="09f5a67bbdeb46778537162bf6753af2-4.18.0-553.22.1.el8_10.x86_64"
    2024-10-16 12:19:10.957 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: index=1
    2024-10-16 12:19:10.958 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: kernel="/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64"
    2024-10-16 12:19:10.959 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: args="ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet systemd.log_level=debug systemd.log_target=kmsg $tuned_params net.ifnames=0"
    2024-10-16 12:19:10.960 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: root="/dev/mapper/rhel-root"
    2024-10-16 12:19:10.961 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: initrd="/boot/initramfs-4.18.0-553.22.1.el8_10.x86_64.img $tuned_initrd"
    2024-10-16 12:19:10.962 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: title="Red Hat Enterprise Linux (4.18.0-553.22.1.el8_10.x86_64) 8.10 (Ootpa) with debugging"
    2024-10-16 12:19:10.963 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: id="09f5a67bbdeb46778537162bf6753af2-4.18.0-553.22.1.el8_10.x86_64~debug"
    2024-10-16 12:19:10.965 DEBUG    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: External command has finished: ['grubby', '--info', '/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64']
    2024-10-16 12:19:10.966 WARNING  PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: Grubby output is malformed: `root=` is listed more than once.
    2024-10-16 12:19:10.967 WARNING  PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: Grubby output is malformed: `args=` is listed more than once.
    2024-10-16 12:19:10.968 ERROR    PID: 2134 leapp.workflow.Finalization.kernelcmdlineconfig: Grubby listed `args=` multiple times with different values.
    2024-10-16 12:19:10.969 WARNING  PID: 2134 leapp.reporting: Stable Key report entry not provided, dynamically generating one - 91f1a3ce231781b3f55244221551e2d206b28a30
----------------------------------------------------------------------

and in the report:

----------------------------------------
Risk Factor: high
Title: Could not set the kernel arguments for future kernels
Summary: During the upgrade we needed to modify the kernel command line arguments. We were able to change the arguments for the default kernel but we were not able to set the arguments as the default for kernels installed in the future.
Remediation: [hint] After the system has been rebooted into the new version of RHEL, you should take the kernel cmdline arguments from /proc/cmdline (Everything except the BOOT_IMAGE entry and initrd entries) and then use the grub2-editenv command to make them the default kernel args.  For example, if /proc/cmdline contains:

    BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-425.3.1.el8.x86_64 root=/dev/mapper/rhel_ibm--root ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH

 then run the following grub2-editenv command:

    # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel_ibm--root ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH"
Key: 91f1a3ce231781b3f55244221551e2d206b28a30
----------------------------------------

Note that in this case, this has been kind of expected behaviour as the process is expecting to see just one bootloader entry for the target kernel and in case that multiple entries exists, we are not sure which value to set. However, thinking about machines with MAKEDEBUG=yes I guess we could do a simple heuristic, when a second entry is always ignored (debug entries are created as next ones after non-debug entries) e.g. when debug parameter is detected. Also the msg about root= listed multiple times with different values is misleading, as in this case values are same and hence it should not be listed at all - a check for other values is not performed. So thinking about that, I think we should update this solution in this PR, to handle it properly. WDYT @matejmatuska ?

matejmatuska commented 1 week ago

Test results from 8->9:

[root@leapp-20241018165724 ~]# grubby --info=ALL
index=0
kernel="/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet $tuned_params"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-4.18.0-553.22.1.el8_10.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-553.22.1.el8_10.x86_64) 8.10 (Ootpa)"
id="fab6ddf4382e451f8fe707758bb8fd2f-4.18.0-553.22.1.el8_10.x86_64"
index=1
kernel="/boot/vmlinuz-4.18.0-553.22.1.el8_10.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet systemd.log_level=debug systemd.log_target=kmsg $tuned_params"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-4.18.0-553.22.1.el8_10.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-553.22.1.el8_10.x86_64) 8.10 (Ootpa) with debugging"
id="fab6ddf4382e451f8fe707758bb8fd2f-4.18.0-553.22.1.el8_10.x86_64~debug"
index=2
kernel="/boot/vmlinuz-4.18.0-553.el8_10.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet $tuned_params"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-4.18.0-553.el8_10.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-553.el8_10.x86_64) 8.10 (Ootpa)"
id="fab6ddf4382e451f8fe707758bb8fd2f-4.18.0-553.el8_10.x86_64"
index=3
kernel="/boot/vmlinuz-4.18.0-527.el8.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet $tuned_params"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-4.18.0-527.el8.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (4.18.0-527.el8.x86_64) 8.10 (Ootpa)"
id="fab6ddf4382e451f8fe707758bb8fd2f-4.18.0-527.el8.x86_64"
index=4
kernel="/boot/vmlinuz-0-rescue-fab6ddf4382e451f8fe707758bb8fd2f"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-0-rescue-fab6ddf4382e451f8fe707758bb8fd2f.img"
title="Red Hat Enterprise Linux (0-rescue-fab6ddf4382e451f8fe707758bb8fd2f) 8.10 (Ootpa)"
id="fab6ddf4382e451f8fe707758bb8fd2f-0-rescue"
index=5
kernel="/boot/vmlinuz-upgrade.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet $tuned_params enforcing=0 plymouth.enable=0 rd.plymouth=0"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-upgrade.x86_64.img"
title="RHEL-Upgrade-Initramfs"
id="fab6ddf4382e451f8fe707758bb8fd2f-upgrade.x86_64"
index=6
kernel="/boot/vmlinuz-upgrade.x86_64"
args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 net.ifnames=0 crashkernel=auto resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet $tuned_params enforcing=0 plymouth.enable=0 rd.plymouth=0 systemd.log_level=debug systemd.log_target=kmsg"
root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
initrd="/boot/initramfs-upgrade.x86_64.img"
title="RHEL-Upgrade-Initramfs with debugging"
id="fab6ddf4382e451f8fe707758bb8fd2f-upgrade.x86_64~debug"

[root@leapp-20241018165724 ~]# leapp upgrade --debug --reboot

From /var/log/leapp/leapp-report.txt

Risk Factor: high
Title: Ensure that expected default kernel cmdline arguments are set
Summary: During the upgrade we needed to modify the kernel command line arguments. However, multiple bootloader entries with different arguments were found for the default kernel (perhaps MAKEDEBUG=yes is set in /etc/sysconfig/kernel). Leapp used the arguments from the first found entry of the target kernel and set it as the new default kernel cmdline arguments for kernels installed in the future.
Remediation: [hint] After the system has been rebooted into the new version of RHEL, check that configured default kernel cmdline arguments in /etc/kernel/cmdline  are correct. In case that different arguments are expected, update the file as needed.

From /var/log/leapp/leapp-upgrade.log:

2024-10-18 15:46:45.952 DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: External command has started: ['grubby', '--info', '/boot/vmlinuz-5.14.0-518.el9.x86_64']
2024-10-18 15:46:46.29  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: index=0
2024-10-18 15:46:46.30  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: kernel="/boot/vmlinuz-5.14.0-518.el9.x86_64"
2024-10-18 15:46:46.31  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet plymouth.enable=0 rd.plymouth=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M $tuned_params net.ifnames=0"
2024-10-18 15:46:46.32  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
2024-10-18 15:46:46.34  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: initrd="/boot/initramfs-5.14.0-518.el9.x86_64.img $tuned_initrd"
2024-10-18 15:46:46.35  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: title="Red Hat Enterprise Linux (5.14.0-518.el9.x86_64) 9.6 (Plow)"
2024-10-18 15:46:46.36  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: id="fab6ddf4382e451f8fe707758bb8fd2f-5.14.0-518.el9.x86_64"
2024-10-18 15:46:46.41  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: index=1
2024-10-18 15:46:46.42  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: kernel="/boot/vmlinuz-5.14.0-518.el9.x86_64"
2024-10-18 15:46:46.43  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: args="ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet plymouth.enable=0 rd.plymouth=0 systemd.log_level=debug systemd.log_target=kmsg crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M $tuned_params net.ifnames=0"
2024-10-18 15:46:46.44  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: root="/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root"
2024-10-18 15:46:46.45  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: initrd="/boot/initramfs-5.14.0-518.el9.x86_64.img $tuned_initrd"
2024-10-18 15:46:46.46  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: title="Red Hat Enterprise Linux (5.14.0-518.el9.x86_64) 9.6 (Plow) with debugging"
2024-10-18 15:46:46.47  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: id="fab6ddf4382e451f8fe707758bb8fd2f-5.14.0-518.el9.x86_64~debug"
2024-10-18 15:46:46.49  DEBUG    PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: External command has finished: ['grubby', '--info', '/boot/vmlinuz-5.14.0-518.el9.x86_64']
2024-10-18 15:46:46.50  WARNING  PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: Grubby output listed `args=` multiple times with different values, continuing with the first result
2024-10-18 15:46:46.52  WARNING  PID: 1851 leapp.workflow.Finalization.kernelcmdlineconfig: Grubby output listed `root=` more than once

[root@leapp-20241018165724 ~]# cat /etc/kernel/cmdline
root=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-root ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH biosdevname=0 resume=/dev/mapper/rhel_ibm--p8--kvm--03--guest--02-swap rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/root rd.lvm.lv=rhel_ibm-p8-kvm-03-guest-02/swap rhgb quiet plymouth.enable=0 rd.plymouth=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M $tuned_params net.ifnames=0

oamg / leapp-repository