package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-3.el8.noarch when upgrade almalinux 8 to 9

Actual behavior I want to update my centos 7 to the latest almalinux 9, now I have updated to almalinux 8. Then I followed the guidance on https://techviewleo.com/how-to-upgrade-to-almalinux-9-from-almalinux-8/. But it failed when I trying to install/update leapp-upgrade.

To Reproduce Steps to reproduce the behavior

execute sudo yum install -y leapp-upgrade leapp-data-almalinux
See error

Expected behavior leapp-upgrade can be install or update successfully.

System information (please complete the following information):

OS and version: AlmaLinux 8.7
# uname -a: Linux localhost.localdomain 4.18.0-425.19.2.el8_7.x86_64 oamg/leapp#1 SMP Tue Apr 4 05:30:47 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa "*leapp*" (or shorthashes of commits in case of manual installation):

[root@localhost apitable]# rpm -qa "*leapp*"
leapp-data-almalinux-0.1-6.el7.noarch
python2-leapp-0.14.0-1.el7.noarch
leapp-0.14.0-1.el7.noarch
leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.8.noarch
leapp-repository-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch
leapp-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch
[root@localhost apitable]#

Shell logs

[root@localhost ~]# curl https://repo.almalinux.org/elevate/testing/elevate-testing.repo -o /etc/yum.repos.d/elevate-testing.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   437  100   437    0     0    520      0 --:--:-- --:--:-- --:--:--   520
[root@localhost ~]# ls /etc/yum.repos.d/ | grep elevate
elevate-testing.repo
[root@localhost ~]# rpm --import https://repo.almalinux.org/elevate/RPM-GPG-KEY-ELevate
[root@localhost ~]# dnf clean all && dnf repolist
[root@localhost ~]#
[root@localhost ~]# yum install -y leapp-upgrade leapp-data-almalinux
CentOS-7 - SCLo rh                                                    226 kB/s |  18 MB     01:21    
AlmaLinux 8 - BaseOS                                                  2.7 MB/s | 6.3 MB     00:02    
AlmaLinux 8 - AppStream                                               2.6 MB/s |  12 MB     00:04    
AlmaLinux 8 - Extras                                                   12 kB/s |  19 kB     00:01    
Docker CE Stable - x86_64                                              38 kB/s |  41 kB     00:01    
ELevate Testing                                                        31 kB/s |  71 kB     00:02    
Extra Packages for Enterprise Linux 8 - x86_64                        4.3 MB/s |  14 MB     00:03    
google-chrome                                                         4.4 kB/s | 3.6 kB     00:00    
MySQL 8.0 Community Server                                            1.0 MB/s | 2.9 MB     00:02    
MySQL Connectors Community                                             50 kB/s |  55 kB     00:01    
MySQL Tools Community                                                 437 kB/s | 925 kB     00:02    
Node.js Packages for Enterprise Linux 7 - x86_64                      146 kB/s | 674 kB     00:04    
Trzsz Repo                                                            2.2 kB/s | 7.3 kB     00:03    
Yarn Repository                                                       272 kB/s | 316 kB     00:01    
Package leapp-data-almalinux-0.1-6.el7.noarch is already installed.
Error: 
 Problem: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch
  - package leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
  - cannot install the best candidate for the job
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost ~]# dnf remove leapp*
Error: 
 Problem 1: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch requires leapp-repository-dependencies = 6, but none of the providers can be installed
  - conflicting requests
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
 Problem 2: package python2-leapp-0.14.0-1.el7.noarch requires leapp-framework-dependencies = 5, but none of the providers can be installed
  - conflicting requests
  - problem with installed package python2-leapp-0.14.0-1.el7.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost ~]# yum remove leapp*
Error: 
 Problem 1: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch requires leapp-repository-dependencies = 6, but none of the providers can be installed
  - conflicting requests
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
 Problem 2: package python2-leapp-0.14.0-1.el7.noarch requires leapp-framework-dependencies = 5, but none of the providers can be installed
  - conflicting requests
  - problem with installed package python2-leapp-0.14.0-1.el7.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost ~]#
[root@localhost ~]# yum install -y leapp-upgrade leapp-data-almalinux
Last metadata expiration check: 2:00:31 ago on Sat 22 Apr 2023 06:19:10 AM CST.
Package leapp-data-almalinux-0.1-6.el7.noarch is already installed.
Error: 
 Problem: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-3.el8.noarch
  - package leapp-upgrade-el8toel9-0.17.0-3.el8.noarch conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
  - conflicting requests
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost ~]# yum install -y leapp-upgrade leapp-data-almalinux --allowerasing
Last metadata expiration check: 2:01:10 ago on Sat 22 Apr 2023 06:19:10 AM CST.
Package leapp-data-almalinux-0.1-6.el7.noarch is already installed.
Error: 
 Problem: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-3.el8.noarch
  - package leapp-upgrade-el8toel9-0.17.0-3.el8.noarch conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
  - conflicting requests
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost ~]# yum install -y leapp-upgrade leapp-data-almalinux --allowerasing --skip-broken
Last metadata expiration check: 2:01:41 ago on Sat 22 Apr 2023 06:19:10 AM CST.
Package leapp-data-almalinux-0.1-6.el7.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost ~]#

I also tried to remove leapp-upgrade package. But it seems could not found this package.

[root@localhost apitable]# sudo yum install -y leapp-upgrade 
Last metadata expiration check: 0:18:40 ago on Sat 22 Apr 2023 10:41:28 AM CST.
Error: 
 Problem: package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch
  - package leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
  - cannot install the best candidate for the job
  - problem with installed package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost apitable]# rpm -qa leapp*
leapp-data-almalinux-0.2-2.el8.noarch
leapp-0.14.0-1.el7.noarch
leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.8.noarch
leapp-repository-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch
leapp-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch
[root@localhost apitable]# yum remove leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.8.noarch
All matches were filtered out by exclude filtering for argument: leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.8.noarch
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost apitable]# yum remove leapp-upgrade-el7toel8
All matches were filtered out by exclude filtering for argument: leapp-upgrade-el7toel8
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost apitable]# yum remove leapp-upgrade
All matches were filtered out by exclude filtering for argument: leapp-upgrade
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost apitable]#
[root@localhost apitable]# rpm -e leapp-upgrade
error: package leapp-upgrade is not installed
[root@localhost apitable]# rpm -e leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.8.noarch
error: Failed dependencies:
    leapp-repository is needed by (installed) leapp-0.14.0-1.el7.noarch
[root@localhost apitable]# rpm -e leapp-0.14.0-1.el7.noarch
error: Failed dependencies:
    leapp is needed by (installed) leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
[root@localhost apitable]# rpm -e leapp-0.14.0-1.el7.noarch leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch
[root@localhost apitable]# sudo yum install -y leapp-upgrade leapp-data-almalinux
Last metadata expiration check: 0:27:11 ago on Sat 22 Apr 2023 10:41:28 AM CST.
Package leapp-data-almalinux-0.2-2.el8.noarch is already installed.
Error: 
 Problem: package leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch requires leapp, but none of the providers can be installed
  - cannot install the best candidate for the job
  - package leapp-0.14.0-1.el8_6.noarch is filtered out by exclude filtering
  - package leapp-0.15.0-2.el8.noarch is filtered out by exclude filtering
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost apitable]# sudo yum install -y leapp leapp-upgrade leapp-data-almalinux
Last metadata expiration check: 0:27:27 ago on Sat 22 Apr 2023 10:41:28 AM CST.
All matches were filtered out by exclude filtering for argument: leapp
Package leapp-data-almalinux-0.2-2.el8.noarch is already installed.
Error: Unable to find a match: leapp
[root@localhost apitable]# sudo yum install -y leapp-0.15.0-2.el8.noarch
Last metadata expiration check: 0:28:16 ago on Sat 22 Apr 2023 10:41:28 AM CST.
All matches were filtered out by exclude filtering for argument: leapp-0.15.0-2.el8.noarch
Error: Unable to find a match: leapp-0.15.0-2.el8.noarch
[root@localhost apitable]# yum update
Last metadata expiration check: 0:28:26 ago on Sat 22 Apr 2023 10:41:28 AM CST.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost apitable]#
[root@localhost apitable]# cat /etc/yum.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp
[root@localhost apitable]# vim /etc/yum.conf # comment exclude
[root@localhost apitable]# cat /etc/yum.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
# exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp
[root@localhost apitable]# sudo yum install -y leapp leapp-upgrade leapp-data-almalinux
Last metadata expiration check: 0:30:51 ago on Sat 22 Apr 2023 10:41:28 AM CST.
Package leapp-data-almalinux-0.2-2.el8.noarch is already installed.
Error: 
 Problem: package leapp-0.14.0-1.el8_6.noarch requires python3-leapp = 0.14.0-1.el8_6, but none of the providers can be installed
  - package python2-leapp-0.14.0-1.el7.noarch conflicts with python3-leapp provided by python3-leapp-0.14.0-1.el8_6.noarch
  - package python3-leapp-0.14.0-1.el8_6.noarch conflicts with python2-leapp provided by python2-leapp-0.14.0-1.el7.noarch
  - cannot install the best candidate for the job
  - problem with installed package python2-leapp-0.14.0-1.el7.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@localhost apitable]#

After I comment exclude in /etc/yum.conf. Now I can remove leapp* and install the new version now.

[root@localhost ~]# vim /etc/yum.conf 
[root@localhost ~]# yum remove leapp*
Dependencies resolved.
=================================================================================================================================
 Package                            Architecture    Version                                              Repository         Size
=================================================================================================================================
Removing:
 leapp                              noarch          0.14.0-1.el7                                         @elevate           60 k
 leapp-data-almalinux               noarch          0.1-6.el7                                            @elevate          1.7 M
 leapp-deps-el8                     noarch          5.0.8-100.202203181036Z.249925a3.master.el8          @System             0  
 leapp-repository-deps-el8          noarch          5.0.8-100.202203181036Z.249925a3.master.el8          @System             0  
 leapp-upgrade-el7toel8             noarch          1:0.16.0-6.el7.elevate.8                             @elevate          2.6 M
Removing dependent packages:
 python2-leapp                      noarch          0.14.0-1.el7                                         @System           628 k

Transaction Summary
=================================================================================================================================
Remove  6 Packages

Freed space: 4.9 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                         1/1 
  Erasing          : leapp-0.14.0-1.el7.noarch                                                                               1/6 
  Erasing          : leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch                                                  2/6 
  Erasing          : python2-leapp-0.14.0-1.el7.noarch                                                                       3/6 
  Erasing          : leapp-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                                       4/6 
  Erasing          : leapp-repository-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                            5/6 
  Erasing          : leapp-data-almalinux-0.1-6.el7.noarch                                                                   6/6 
  Running scriptlet: leapp-data-almalinux-0.1-6.el7.noarch                                                                   6/6 
  Verifying        : leapp-0.14.0-1.el7.noarch                                                                               1/6 
  Verifying        : leapp-data-almalinux-0.1-6.el7.noarch                                                                   2/6 
  Verifying        : leapp-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                                       3/6 
  Verifying        : leapp-repository-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                            4/6 
  Verifying        : leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch                                                  5/6 
  Verifying        : python2-leapp-0.14.0-1.el7.noarch                                                                       6/6 

Removed:
  leapp-0.14.0-1.el7.noarch                                                                                                      
  leapp-data-almalinux-0.1-6.el7.noarch                                                                                          
  leapp-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                                                              
  leapp-repository-deps-el8-5.0.8-100.202203181036Z.249925a3.master.el8.noarch                                                   
  leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch                                                                         
  python2-leapp-0.14.0-1.el7.noarch                                                                                              

Complete!
[root@localhost ~]#
[root@localhost ~]# yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el8.noarch.rpm
Last metadata expiration check: 1:37:58 ago on Sat 22 Apr 2023 01:13:05 PM CST.
elevate-release-latest-el8.noarch.rpm                                                            7.7 kB/s | 9.8 kB     00:01    
Dependencies resolved.
=================================================================================================================================
 Package                            Architecture              Version                      Repository                       Size
=================================================================================================================================
Upgrading:
 elevate-release                    noarch                    1.0-2.el8                    @commandline                    9.8 k

Transaction Summary
=================================================================================================================================
Upgrade  1 Package

Total size: 9.8 k
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                         1/1 
  Upgrading        : elevate-release-1.0-2.el8.noarch                                                                        1/2 
  Cleanup          : elevate-release-1.0-2.el7.noarch                                                                        2/2 
  Verifying        : elevate-release-1.0-2.el8.noarch                                                                        1/2 
  Verifying        : elevate-release-1.0-2.el7.noarch                                                                        2/2 

Upgraded:
  elevate-release-1.0-2.el8.noarch                                                                                               

Complete!
[root@localhost ~]#
[root@localhost ~]# yum install -y leapp-upgrade leapp-data-almalinux
Last metadata expiration check: 1:38:16 ago on Sat 22 Apr 2023 01:13:05 PM CST.
Dependencies resolved.
=================================================================================================================================
 Package                                 Architecture       Version                                    Repository           Size
=================================================================================================================================
Installing:
 leapp-data-almalinux                    noarch             0.2-2.el8                                  elevate              57 k
 leapp-upgrade-el8toel9                  noarch             1:0.16.0-6.el8_6.elevate.10                elevate             639 k
Installing dependencies:
 leapp                                   noarch             0.14.0-1.el8_6                             elevate              31 k
 leapp-deps                              noarch             0.14.0-1.el8_6                             elevate              14 k
 leapp-upgrade-el8toel9-deps             noarch             1:0.16.0-6.el8_6.elevate.10                elevate              28 k
 python3-leapp                           noarch             0.14.0-1.el8_6                             elevate             174 k

Transaction Summary
=================================================================================================================================
Install  6 Packages

Total download size: 943 k
Installed size: 3.9 M
Downloading Packages:
(1/6): leapp-0.14.0-1.el8_6.noarch.rpm                                                            27 kB/s |  31 kB     00:01    
(2/6): leapp-deps-0.14.0-1.el8_6.noarch.rpm                                                       12 kB/s |  14 kB     00:01    
(3/6): leapp-data-almalinux-0.2-2.el8.noarch.rpm                                                  39 kB/s |  57 kB     00:01    
(4/6): leapp-upgrade-el8toel9-deps-0.16.0-6.el8_6.elevate.10.noarch.rpm                           50 kB/s |  28 kB     00:00    
(5/6): python3-leapp-0.14.0-1.el8_6.noarch.rpm                                                   203 kB/s | 174 kB     00:00    
(6/6): leapp-upgrade-el8toel9-0.16.0-6.el8_6.elevate.10.noarch.rpm                               456 kB/s | 639 kB     00:01    
---------------------------------------------------------------------------------------------------------------------------------
Total                                                                                            366 kB/s | 943 kB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                         1/1 
  Installing       : leapp-upgrade-el8toel9-deps-1:0.16.0-6.el8_6.elevate.10.noarch                                          1/6 
  Installing       : leapp-deps-0.14.0-1.el8_6.noarch                                                                        2/6 
  Installing       : python3-leapp-0.14.0-1.el8_6.noarch                                                                     3/6 
  Installing       : leapp-0.14.0-1.el8_6.noarch                                                                             4/6 
  Installing       : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                               5/6 
  Installing       : leapp-data-almalinux-0.2-2.el8.noarch                                                                   6/6 
  Running scriptlet: leapp-data-almalinux-0.2-2.el8.noarch                                                                   6/6 
  Verifying        : leapp-0.14.0-1.el8_6.noarch                                                                             1/6 
  Verifying        : leapp-data-almalinux-0.2-2.el8.noarch                                                                   2/6 
  Verifying        : leapp-deps-0.14.0-1.el8_6.noarch                                                                        3/6 
  Verifying        : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                               4/6 
  Verifying        : leapp-upgrade-el8toel9-deps-1:0.16.0-6.el8_6.elevate.10.noarch                                          5/6 
  Verifying        : python3-leapp-0.14.0-1.el8_6.noarch                                                                     6/6 

Installed:
  leapp-0.14.0-1.el8_6.noarch                                       leapp-data-almalinux-0.2-2.el8.noarch                       
  leapp-deps-0.14.0-1.el8_6.noarch                                  leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch   
  leapp-upgrade-el8toel9-deps-1:0.16.0-6.el8_6.elevate.10.noarch    python3-leapp-0.14.0-1.el8_6.noarch                         

Complete!
[root@localhost ~]#
[root@localhost ~]# cat /etc/yum.conf 
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
# exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp
[root@localhost ~]#

Now I have updated from almalinux 8 to 9, during the process, I have to remove some packages like nodejs and so on, and fix some other problems like comment modprobe xen-hcall >/dev/null 2>&1 line of /etc/rc.d/rc.local or I will got Failed to start Load Kernel Modules warning.

And the networking configuration is missing, and I have to configure again. The method of config network is also changed, see https://www.redhat.com/en/blog/rhel-9-networking-say-goodbye-ifcfg-files-and-hello-keyfiles, https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_and_managing_networking/index.

@liudonghua123 Hi \o the manual removal of leapp packages from the exclude list is expected before they can be removed from the system is expected step, supposed to be done manually. I am not familiar with the almalinux documentation - in case it's not documented, you need to contact almalinux support for the improvement.

Now I have updated from almalinux 8 to 9, during the process, I have to remove some packages like nodejs and so on

We have no knowledge about almalinux packages. You need to contact almalinux to let them know about the problem so they can fix it. Most likely they need to update PES data.

and fix some other problems like comment modprobe xen-hcall >/dev/null 2>&1 line of /etc/rc.d/rc.local or I will got Failed to start Load Kernel Modules warning.

what package provides that module?

And the networking configuration is missing, and I have to configure again. The method of config network is also changed, see https://www.redhat.com/en/blog/rhel-9-networking-say-goodbye-ifcfg-files-and-hello-keyfiles, https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_and_managing_networking/index.

Can you share the generated report file? /var/log/leapp/leapp-report.txt

Also, from the output I see you used and old version of leapp-upgrade-el8toel9 package. Regarding network scripts, we have provided number of improvements in the current upstream version of leapp-repository 0.18.0 . It would help us also to know the behaviour with this version. If still not present in almalinux, possibly it will be introduced in couple of weeks/months. In case the same problem is still present, we will need then more data about the system, produced leapp messages, logs,....

@lkundrak ping - letting you know ^

@pirat89 Thanks for your detailed explanation, here is my last leapp-report.txt and leapp-upgrade.log.

leapp-upgrade.log

leapp-20230422155746-logs.tar.gz

@liudonghua123 Problems you see on RHEL 9 after the upgrade are caused by failed & unfinished upgrade process:

56078 Apr 22 08:32:16 localhost upgrade[74628]: Traceback (most recent call last):    
56079 Apr 22 08:32:16 localhost upgrade[74628]:   File "/root/tmp_leapp_py3/leapp3", line 6, in <module>
56080 Apr 22 08:32:16 localhost upgrade[74628]:     sys.exit(leapp.cli.main())        
56081 Apr 22 08:32:16 localhost upgrade[74628]: AttributeError: module 'leapp.cli' has no attribute 'main'

Not sure what happened on the system - I see this error for the first time in all those years so not sure whether it's something what is connected to our code or whether it's a problem connected to a difference between rhel and almalinux. In short, all "post-upgrade" phases / actors have not been executed. It's possible that more things could be broken on the system - like the networking. I guess that your networking setup has been expected to be really migrated automatically as I thought, but it has not been due to the problem above.

Also I see that leapp has been executed possibly again - trying to perform the upgrade transaction one more time?:

Transaction Summary
=============================================================================================================================
Install     255 Packages
Upgrade    1535 Packages
Remove      181 Packages
Downgrade    46 Packages

Total size: 1.8 G
Total download size: 32 M
Downloading Packages:

STDERR:
No matches found for the following disable plugin patterns: subscription-manager
Repository appstream is listed more than once in the configuration
Repository appstream-source is listed more than once in the configuration
Repository baseos is listed more than once in the configuration
Repository baseos-source is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository extras-source is listed more than once in the configuration
Error opening /var/cache/dnf/almalinux9-appstream-0a8174e0a3d29b90/packages/nodejs-16.18.1-3.el9_1.x86_64.rpm: No such file or directory
Package "nodejs-1:16.18.1-3.el9_1.x86_64" from repository "almalinux9-appstream" has incorrect checksum
Error opening /var/cache/dnf/almalinux9-appstream-0a8174e0a3d29b90/packages/nodejs-docs-16.18.1-3.el9_1.noarch.rpm: No such file or directory
Package "nodejs-docs-1:16.18.1-3.el9_1.noarch" from repository "almalinux9-appstream" has incorrect checksum
Error opening /var/cache/dnf/almalinux9-appstream-0a8174e0a3d29b90/packages/nodejs-full-i18n-16.18.1-3.el9_1.x86_64.rpm: No such file or directory
Package "nodejs-full-i18n-1:16.18.1-3.el9_1.x86_64" from repository "almalinux9-appstream" has incorrect checksum
Error opening /var/cache/dnf/almalinux9-appstream-0a8174e0a3d29b90/packages/npm-8.19.2-1.16.18.1.3.el9_1.x86_64.rpm: No such file or directory
Package "npm-1:8.19.2-1.16.18.1.3.el9_1.x86_64" from repository "almalinux9-appstream" has incorrect checksum
Error opening /var/cache/dnf/almalinux9-appstream-0a8174e0a3d29b90/packages/nodejs-libs-16.18.1-3.el9_1.x86_64.rpm: No such file or directory
Package "nodejs-libs-1:16.18.1-3.el9_1.x86_64" from repository "almalinux9-appstream" has incorrect checksum
Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option

This could happen if an error like the first one happens, so there is the loop when leapp is starting the process inside the upgrade environment again in "a loop" - not 100% it's this case but it could explain the 2nd issue. This is also fixed in the latest upstream release. But the original issue remians mysterious to me.

Thanks, maybe the version of leapp is too old in almalinux repository. I will try to keep an eye on the upgrade process later.

The network configuration like you said is processed in leapp.cli.main() which is not available on my system. I don't know why and couldn't find any methods to investigate it. Is a normal python package installed via pip or some python scripts located somewhere on the system?

I also checked/reinstalled the leapp on my system.

[root@localhost ~]# cat /etc/os-release 
NAME="AlmaLinux"
VERSION="9.1 (Lime Lynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.1 (Lime Lynx)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
[root@localhost ~]# uname -a
Linux localhost.localdomain 5.14.0-162.23.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 10:43:28 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# 
[root@localhost ~]# which leapp
/usr/bin/leapp
[root@localhost ~]# file /usr/bin/leapp
/usr/bin/leapp: Python script, ASCII text executable
[root@localhost ~]# vim /usr/bin/leapp
[root@localhost ~]# leapp --version
Traceback (most recent call last):
  File "/usr/bin/leapp", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'leapp==0.14.0' distribution was not found and is required by the application
[root@localhost ~]# cat /usr/bin/leapp 
#!/usr/libexec/platform-python
# EASY-INSTALL-ENTRY-SCRIPT: 'leapp==0.14.0','console_scripts','leapp'
__requires__ = 'leapp==0.14.0'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('leapp==0.14.0', 'console_scripts', 'leapp')()
    )
[root@localhost ~]# rpm -qa leapp*
[root@localhost ~]# yum install leapp
Last metadata expiration check: 2:28:48 ago on Mon 24 Apr 2023 06:22:08 PM CST.
All matches were filtered out by exclude filtering for argument: leapp
Error: Unable to find a match: leapp
[root@localhost ~]# vim /etc/yum.conf 
[root@localhost ~]# yum install leapp
Last metadata expiration check: 2:29:18 ago on Mon 24 Apr 2023 06:22:08 PM CST.
Package leapp-0.14.0-1.el8_6.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost ~]# yum reinstall leapp
Last metadata expiration check: 2:29:43 ago on Mon 24 Apr 2023 06:22:08 PM CST.
Dependencies resolved.
=========================================================================================================================================================================================
 Package                                  Architecture                              Version                                             Repository                                  Size
=========================================================================================================================================================================================
Reinstalling:
 leapp                                    noarch                                    0.14.0-1.el8_6                                      elevate                                     31 k

Transaction Summary
=========================================================================================================================================================================================

Total download size: 31 k
Installed size: 60 k
Is this ok [y/N]: y
Downloading Packages:
leapp-0.14.0-1.el8_6.noarch.rpm                                                                                                                           20 kB/s |  31 kB     00:01    
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                     16 kB/s |  31 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                 1/1 
  Reinstalling     : leapp-0.14.0-1.el8_6.noarch                                                                                                                                     1/2 
  Cleanup          : leapp-0.14.0-1.el8_6.noarch                                                                                                                                     2/2 
  Running scriptlet: leapp-0.14.0-1.el8_6.noarch                                                                                                                                     2/2 
  Verifying        : leapp-0.14.0-1.el8_6.noarch                                                                                                                                     1/2 
  Verifying        : leapp-0.14.0-1.el8_6.noarch                                                                                                                                     2/2 
Installed products updated.

Reinstalled:
  leapp-0.14.0-1.el8_6.noarch                                                                                                                                                            

Complete!
[root@localhost ~]# leapp --version
Traceback (most recent call last):
  File "/usr/bin/leapp", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'leapp==0.14.0' distribution was not found and is required by the application
[root@localhost ~]# cat /etc/yum.conf 
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
# exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp
# exclude=leapp,python3-leapp,snactor,leapp-upgrade-el8toel9
[root@localhost ~]# yum install leapp-upgrade python3-leapp
Last metadata expiration check: 2:30:49 ago on Mon 24 Apr 2023 06:22:08 PM CST.
Package leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch is already installed.
Package python3-leapp-0.14.0-1.el8_6.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@localhost ~]# yum reinstall leapp-upgrade python3-leapp
Last metadata expiration check: 2:31:00 ago on Mon 24 Apr 2023 06:22:08 PM CST.
Dependencies resolved.
=========================================================================================================================================================================================
 Package                                            Architecture                       Version                                                 Repository                           Size
=========================================================================================================================================================================================
Reinstalling:
 leapp-upgrade-el8toel9                             noarch                             1:0.16.0-6.el8_6.elevate.10                             elevate                             639 k
 python3-leapp                                      noarch                             0.14.0-1.el8_6                                          elevate                             174 k

Transaction Summary
=========================================================================================================================================================================================

Total download size: 813 k
Installed size: 2.1 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): python3-leapp-0.14.0-1.el8_6.noarch.rpm                                                                                                            94 kB/s | 174 kB     00:01    
(2/2): leapp-upgrade-el8toel9-0.16.0-6.el8_6.elevate.10.noarch.rpm                                                                                       285 kB/s | 639 kB     00:02    
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                    360 kB/s | 813 kB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                 1/1 
  Reinstalling     : python3-leapp-0.14.0-1.el8_6.noarch                                                                                                                             1/4 
  Reinstalling     : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                                                                                       2/4 
  Cleanup          : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                                                                                       3/4 
  Cleanup          : python3-leapp-0.14.0-1.el8_6.noarch                                                                                                                             4/4 
  Running scriptlet: python3-leapp-0.14.0-1.el8_6.noarch                                                                                                                             4/4 
  Verifying        : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                                                                                       1/4 
  Verifying        : leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                                                                                       2/4 
  Verifying        : python3-leapp-0.14.0-1.el8_6.noarch                                                                                                                             3/4 
  Verifying        : python3-leapp-0.14.0-1.el8_6.noarch                                                                                                                             4/4 
Installed products updated.

Reinstalled:
  leapp-upgrade-el8toel9-1:0.16.0-6.el8_6.elevate.10.noarch                                              python3-leapp-0.14.0-1.el8_6.noarch                                             

Complete!
[root@localhost ~]# leapp --version
Traceback (most recent call last):
  File "/usr/bin/leapp", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'leapp==0.14.0' distribution was not found and is required by the application
[root@localhost ~]# python -c "import leapp"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'leapp'
[root@localhost ~]#

It's installed via rpm packages. leapp is just a framework, the inplace upgrade and all actions are defined by leapp-repository (leapp-upgrade-elXtoelY rpm, replace X & Y by major versions of rhel for the upgrade). After the RPM transaction is performed, leapp must be via a special executable created just to finish the upgrade (which will fail now, read the whole comment before experimenting):

/usr/bin/python3 -B /root/tmp_leapp_py3/leapp3 ....

And in case that no other leapp execution has been done yet, "possibly" you could still call leapp to finish the upgrade, using:

/usr/bin/python3 -B /root/tmp_leapp_py3/leapp3 upgrade --resume

But it can be risky to do it now!!! In case you are already booted in the system and did a configuration changes, I think that you want to probably rather fix other issues over time when you hit them, instead of trying leapp to do all changes. E.g. I suggest to check changes in selinux (I would definitely do autorelabel of the system to fix all contexts and then check audit reports). Also possibly custom selinux rules are dropped. Not sure..

Just right now, I got an idea what's possibly wrong. What is output of the following command?

ls -l /root/tmp_leapp_py3

Thinking about that again, it's possible the symlink you find inside is wrong - pointing to an old el7 version of python instead el8 one. The old leapp-repository version had an issue that if people did upgrade 7 -> 8 before, they had to remove this directory manually before upgrading to el9, as there has been a bug. You should see output like this:

[root@localhost ~]# ls -l /root/tmp_leapp_py3/
total 4
lrwxrwxrwx. 1 root root  38 Apr 24 15:49 leapp -> /usr/lib/python3.6/site-packages/leapp
-rwxrwx---. 1 root root 118 Apr 24 15:49 leapp3

And the executable should look like this:

[root@localhost ~]# cat /root/tmp_leapp_py3/leapp3 
#!/usr/bin/python3 -B
import sys
sys.path.append('/root/tmp_leapp_py3')

import leapp.cli
sys.exit(leapp.cli.main())

If you see the symlink pointing to python2, then you hit the old bug that's fixed for a longer time. We had that covered in the official RHEL documentation for IPU 8 -> 9 (th first step; we will drop it soon as it's obsoleted for us) and also up-to-date leapp-repository packages handle this problem (if I remember right, leapp-repository-0-17.0 and newer). I am nof familiar with almalinux documentation, but in case you hit the issue and they do not document it, I suggest to ask them for the fix of the documentation or update of packages.

In case that was the problem you hit, the only issue from the original report that remains is xen-hcall kernel module. Can you tell me what rpm provides / provided that module? Is it provided by the distributioun or is it third-party?

Yes, I updated from centos 7 to almalinux 8 before update to almalinux 9.

[root@localhost ~]# ls -l /root/tmp_leapp_py3
total 4
lrwxrwxrwx 1 root root  38 Apr 21 12:26 leapp -> /usr/lib/python2.7/site-packages/leapp
-rwxrwx--- 1 root root 115 Apr 22 16:32 leapp3
[root@localhost ~]# cat /root/tmp_leapp_py3
cat: /root/tmp_leapp_py3: Is a directory
[root@localhost ~]# cat /root/tmp_leapp_py3/leapp
cat: /root/tmp_leapp_py3/leapp: Is a directory
[root@localhost ~]# file /root/tmp_leapp_py3/leapp3
/root/tmp_leapp_py3/leapp3: Python script, ASCII text executable
[root@localhost ~]# cat /root/tmp_leapp_py3/leapp3
#!/usr/bin/python3
import sys
sys.path.append('/root/tmp_leapp_py3')

import leapp.cli
sys.exit(leapp.cli.main())

[root@localhost ~]#

From the doc you mentioned, I should perform rm -rf /root/tmp_leapp_py3 before update almalinux 8 to 9.

Exactly. That should fix the main problem. Also not sure what else is documented in almalinux, but I suggest to also follow required post-upgrade steps chapter which informs you what is usually required people to do manually after the upgrade to finish it.

@pirat89 have you considered adding Obsoletes for leapp-upgrade-el7toel8 in leapp-upgrade-el8toel9 package? Or this will break things?

@andrewlukoshko we was considered and we want the exact opposite - that's why we have also set Conflict instead and add our packages into the exclude list in dnf configuration on top of that. It must not happen in any state of the leapp run that our packages are updated automatically - as that's leads effectively to the crash of the upgrade process.

Regarding the upgrade process, the Obsoletes does not make sense then as if users finish all required post-upgrade steps, they will never need these two packages to be replaced using dnf update, as part of the postupgrade process is manual removal of all original system packages - including removal of all leapp packages, kernel, .... etc.

Closing the issue as everything has been already discussed and we do not have any additional info about the xen-hcall kmodule, which I consider anyway a different topic.

oamg / leapp-repository

package leapp-upgrade-el7toel8-1:0.16.0-6.el7.elevate.8.noarch conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-3.el8.noarch when upgrade almalinux 8 to 9 #1072