RHEL: Removing subscription certificates too early ?

[kedare]

Hello

I am convertying a personal NAS from RHEL 8 to AlmaLinux 8 using this tool.

During the initial conversion, a step would fail, likely because the subscription manager and the subscription files have been removed too early in the process:

Run dnf distro-sync -y                                                OK
Restoring of alternatives is done                                     OK
Generating grub configuration file ...
File descriptor 5 (/var/log/almalinux-deploy.debug.log) leaked on vgs invocation. Parent PID 50799: /usr/sbin/grub2-probe
File descriptor 5 (/var/log/almalinux-deploy.debug.log) leaked on vgs invocation. Parent PID 50799: /usr/sbin/grub2-probe
File descriptor 5 (/var/log/almalinux-deploy.debug.log) leaked on vgs invocation. Parent PID 50977: /usr/sbin/grub2-probe
File descriptor 5 (/var/log/almalinux-deploy.debug.log) leaked on vgs invocation. Parent PID 50977: /usr/sbin/grub2-probe
device-mapper: reload ioctl on osprober-linux-sdb1 (253:2) failed: Device or resource busy
Command failed.
device-mapper: reload ioctl on osprober-linux-sdc1 (253:2) failed: Device or resource busy
Command failed.
device-mapper: reload ioctl on osprober-linux-sdd1 (253:2) failed: Device or resource busy
Command failed.
done
prometheus                                      277  B/s | 833  B     00:03
prometheus-source                               298  B/s | 819  B     00:02
Dependencies resolved.
================================================================================
 Package     Arch   Version                 Repository                     Size
================================================================================
Reinstalling:
 kernel-core x86_64 4.18.0-477.21.1.el8_8   rhel-8-for-x86_64-baseos-rpms  42 M

Transaction Summary
================================================================================

Total download size: 42 M
Installed size: 70 M
Downloading Packages:
[MIRROR] kernel-core-4.18.0-477.21.1.el8_8.x86_64.rpm: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/Packages/k/kernel-core-4.18.0-477.21.1.el8_8.x86_64.rpm [error setting certificate verify locations:
  CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none]
[FAILED] kernel-core-4.18.0-477.21.1.el8_8.x86_64.rpm: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/Packages/k/kernel-core-4.18.0-477.21.1.el8_8.x86_64.rpm [error setting certificate verify locations:
  CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none]

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  kernel-core-4.18.0-477.21.1.el8_8.x86_64: Download failed: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/Packages/k/kernel-core-4.18.0-477.21.1.el8_8.x86_64.rpm [error setting certificate verify locations:
    CAfile: /etc/rhsm/ca/redhat-uep.pem
    CApath: none]
[root@frnte1-nas1 ~]#

My workaround was to disable those repositories and then rerun the script to be able to continue:

yum-config-manager --disable rhel-8-for-x86_64-appstream-rpms
yum-config-manager --disable rhel-8-for-x86_64-baseos-rpms
yum-config-manager --disable codeready-builder-for-rhel-8-x86_64-rpms

[asclepiadae]

I experienced the same frustration, except I manually removed those repositories before re-running the script, and I'm pretty sure that broke things somewhat spectacularly. Somewhat recovered, but ya - the certificates for RHEL are definitely not removed at the right point.

I think this probably relates to issue #160 and pull #167

Side bar:

The script was completed successfully and the machine rebooted using the AlmaLinux kernel, but persistent errors said a subscription file was missing, couldn't be opened etc. I eventually cleared those errors, but now have an alert that the live kernel patcher process fails, and I seem to have two different versions of the Alma kernel on the machine (although booting with only one). Still trying to solve this issue. Might post to a community discussion here soon.

[yuravk]

Hello,

Thank you for reporting the issue.

A. In general, the problem does not exist if Red Hat Subscription Management is working as expected. The almalinux-deploy utility does not remove anything related to RHSM. It just disables product-id, subscription-manager and upload-profile plugins only.

# grep -r enabled= /etc/dnf/plugins/*
/etc/dnf/plugins/debuginfo-install.conf:enabled=1
/etc/dnf/plugins/product-id.conf:enabled=0
/etc/dnf/plugins/subscription-manager.conf:enabled=0
/etc/dnf/plugins/upload-profile.conf:enabled=0

B. The issue can be reproduced if the following conditions are meat:

1. The kernel version AlmaLinux provides is older then currently installed (provided by Red Hat) on system;
2. Red Hat's system certificate /etc/rhsm/ca/redhat-uep.pem is corrupted or missed.

The first is quite possible, as there is a slight delay in the releases of AlmaLinux kernel corresponding to the latest version from Red Hat. When the second is a kind of system misconfiguration. Please see the log below, with a case the certificate file is manually removed:

Run dnf distro-sync -y                                                OK
Restoring of alternatives is done                                     OK
Generating grub configuration file ...
done
Last metadata expiration check: 0:03:00 ago on Thu 28 Dec 2023 11:38:32 AM EET.
Dependencies resolved.
================================================================================
 Package     Arch   Version                 Repository                     Size
================================================================================
Reinstalling:
 kernel-core x86_64 4.18.0-513.9.1.el8_9    rhel-8-for-x86_64-baseos-rpms  43 M

Transaction Summary
================================================================================

Total download size: 43 M
Installed size: 71 M
Downloading Packages:
[MIRROR] kernel-core-4.18.0-513.9.1.el8_9.x86_64.rpm: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/Packages/k/kernel-core-4.18.0-513.9.1.el8_9.x86_64.rpm [error setting certificate verify locations:
  CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none]
[FAILED] kernel-core-4.18.0-513.9.1.el8_9.x86_64.rpm: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/Packages/k/kernel-core-4.18.0-513.9.1.el8_9.x86_64.rpm [error setting certificate verify locations:
  CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none]
...

# subscription-manager list
System certificates corrupted. Please reregister.
# rpm -qV subscription-manager-rhsm-certificates
missing     /etc/rhsm/ca/redhat-uep.pem
# ls -la /etc/rhsm/ca/redhat-uep.pem
ls: cannot access '/etc/rhsm/ca/redhat-uep.pem': No such file or directory

[yuravk]

The reason of the issue seems to be missed or corrupted RHSM certificate file /etc/rhsm/ca/redhat-uep.pem. So, the point here is to check RHSM works or disable it before deployment.