oamg / leapp-repository

Leapp repositories containing actors for the Leapp framework (https://github.com/oamg/leapp). Currently provides leapp repositories for in-place upgrades of RHEL systems.
Apache License 2.0
52 stars 146 forks source link

Package "X" from local repository "Y" has incorrect checksum #1203

Closed verdverm closed 6 months ago

verdverm commented 6 months ago

Actual behavior

Seeing the following error at the last install step before reboot. It only occurs when a repo mirror is created and unpackaged on the machine being upgraded. The context is an airgapped upgrade, so no internet access is possible.

Install    376 Packages
Upgrade    484 Packages
Remove      78 Packages
Downgrade   10 Packages

Total size: 1.5 G
Total download size: 1.5 G
DNF will only download packages, install gpg keys, and check the transaction.
Downloading Packages:
2024-04-17 15:54:35.258 ERROR    PID: 349 leapp.workflow.Download.dnf_package_download: DNF execution failed: 
2024-04-17 15:54:35.507 ERROR    PID: 24004 leapp: 2024-04-17 15:54:35.444809 [ERROR] Actor: dnf_package_download
Message: DNF execution failed with non zero exit code.

...

Package "zip-3.0-23.el8.x86_64" from local repository "Y-r8-mig" has incorrect checksum
Error opening /media/Y/repos/minimal-r8-mig/zlib-1.2.11-25.el8.x86_64.rpm: No such file or directory
Package "zlib-1.2.11-25.el8.x86_64" from local repository "Y-r8-mig" has incorrect checksum
Error opening /media/Y/repos/minimal-r8-mig/zlib-devel-1.2.11-25.el8.x86_64.rpm: No such file or directory
Package "zlib-devel-1.2.11-25.el8.x86_64" from local repository "Y-r8-mig" has incorrect checksum
Error: Some packages from local repository have incorrect checksum

Note, some of these packages are installed earlier from this local mirror

To Reproduce Steps to reproduce the behavior

  1. reposync & createrepo for rocky-8 repos from another rocky-8 VM, package as tar
  2. unpackage the mirror(s) on the target machine
  3. modify /etc/leapp/files/leapp_upgrade_repositories.repo to point at the location on vm
  4. run LEAPP_NOGPGCHECK=1 leapp upgrade --debug
  5. See error

Expected behavior

Packages should install as usual.

Note, this process does work if we install from our cloud mirror, rather than the local filesystem. So there is something to narrow down where this might be breaking down.

System information (please complete the following information):

# rpm -qa "leapp*"
leapp-0.14.0-1.el7.noarch
leapp-deps-0.14.0-1.el7.noarch
leapp-upgrade-el7toel8-deps-0.16.0-6.el7.elevate.19.noarch
leapp-upgrade-el7toel8-0.16.0-6.el7.elevate.19.noarch
leapp-data-rocky-0.2-5.el7.1.noarch

Attach (or provide link to) log files if applicable (optional - may contain confidential information):

For your convenience you can pack all logs with this command:

# tar -czf leapp-logs.tar.gz /var/log/leapp /var/lib/leapp/leapp.db

Then you may attach only the leapp-logs.tar.gz file.


Additional context

The repo config file for leapp

[Y-r8-mig]
name=Rocky Upgrade Repo
baseurl=file:///media/Y/repos/r8-mig
gpgcheck=0
enabled=1

Debug logs from the failing command

This line stands out to me, as we tried to disable the gpgcheck everywhere possible

2024-04-17 16:07:49.481 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: gpgcheck = 1

logs...

2024-04-17 16:07:49.176 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: External command has started: ['systemd-nspawn', '--register=no', '--quiet', '-D', u'/var/lib/leapp/el8userspace', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/wp-toolkit/files:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el7toel8/files', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=8.6', '--setenv=LEAPP_NO_RHSM=1', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/cloudlinux/tools:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el7toel8/tools', '--setenv=LEAPP_NOGPGCHECK=1', '--setenv=LEAPP_IPU_IN_PROGRESS=7to8', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_EXECUTION_ID=0ba998a0-72ed-4eed-9bd5-2fa106e678d6', '--setenv=LEAPP_HOSTNAME=r8-mig-tony.c.Y-dev.internal', 'dnf', 'config-manager', '--dump']
2024-04-17 16:07:49.424 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: ===================================== main =====================================
2024-04-17 16:07:49.427 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: [main]
2024-04-17 16:07:49.428 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: allow_vendor_change = 1
2024-04-17 16:07:49.430 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: assumeno = 0
2024-04-17 16:07:49.431 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: assumeyes = 0
2024-04-17 16:07:49.433 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: autocheck_running_kernel = 1
2024-04-17 16:07:49.434 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: bandwidth = 0
2024-04-17 16:07:49.436 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: best = 1
2024-04-17 16:07:49.437 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: bugtracker_url = https://bugs.rockylinux.org/
2024-04-17 16:07:49.439 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: cachedir = /var/cache/dnf
2024-04-17 16:07:49.440 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: cacheonly = 0
2024-04-17 16:07:49.442 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: check_config_file_age = 1
2024-04-17 16:07:49.443 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: clean_requirements_on_remove = 1
2024-04-17 16:07:49.445 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color = auto
2024-04-17 16:07:49.446 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_available_downgrade = magenta
2024-04-17 16:07:49.448 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_available_install = bold,cyan
2024-04-17 16:07:49.449 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_available_reinstall = bold,underline,green
2024-04-17 16:07:49.451 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_available_upgrade = bold,blue
2024-04-17 16:07:49.453 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_installed_extra 
= bold,red
2024-04-17 16:07:49.454 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_installed_newer 
= bold,yellow
2024-04-17 16:07:49.456 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_installed_older 
= yellow
2024-04-17 16:07:49.457 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_list_installed_reinst
all = cyan
2024-04-17 16:07:49.458 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_search_match = bold,m
agenta
2024-04-17 16:07:49.459 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_update_installed = re
d
2024-04-17 16:07:49.461 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_update_local = green
2024-04-17 16:07:49.462 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: color_update_remote = bold,
green
2024-04-17 16:07:49.463 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: config_file_path = /etc/dnf
/dnf.conf
2024-04-17 16:07:49.464 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: countme = 0
2024-04-17 16:07:49.465 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: debug_solver = 0
2024-04-17 16:07:49.466 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: debuglevel = 2
2024-04-17 16:07:49.467 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: defaultyes = 0
2024-04-17 16:07:49.469 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: deltarpm = 1
2024-04-17 16:07:49.470 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: deltarpm_percentage = 75
2024-04-17 16:07:49.471 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: disable_excludes = 
2024-04-17 16:07:49.472 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: diskspacecheck = 1
2024-04-17 16:07:49.473 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: enabled = 1
2024-04-17 16:07:49.474 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: enablegroups = 1
2024-04-17 16:07:49.475 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: errorlevel = 3
2024-04-17 16:07:49.476 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: exclude = python2-leapp, sn
actor, leapp-upgrade-el7toel8, leapp
2024-04-17 16:07:49.478 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: excludepkgs = python2-leapp
, snactor, leapp-upgrade-el7toel8, leapp
2024-04-17 16:07:49.479 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: exit_on_lock = 0
2024-04-17 16:07:49.480 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: fastestmirror = 0
2024-04-17 16:07:49.481 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: gpgcheck = 1
2024-04-17 16:07:49.482 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: gpgkey_dns_verification = 0
2024-04-17 16:07:49.483 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: group_package_types = manda
tory, default, conditional
2024-04-17 16:07:49.485 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: history_list_view = command
s
2024-04-17 16:07:49.486 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: history_record = 1
2024-04-17 16:07:49.487 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: history_record_packages = d
nf, rpm
2024-04-17 16:07:49.488 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: ignorearch = 0
2024-04-17 16:07:49.489 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: includepkgs = 
2024-04-17 16:07:49.490 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: install_weak_deps = 1
2024-04-17 16:07:49.491 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: installonly_limit = 3
2024-04-17 16:07:49.493 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: installonlypkgs = kernel, k
ernel-PAE, installonlypkg(kernel), installonlypkg(kernel-module), installonlypkg(vm), multiversion(kernel)
2024-04-17 16:07:49.494 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: installroot = /
2024-04-17 16:07:49.495 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: ip_resolve = whatever
2024-04-17 16:07:49.496 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: keepcache = 0
2024-04-17 16:07:49.497 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: localpkg_gpgcheck = 0
2024-04-17 16:07:49.498 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: log_compress = 0
2024-04-17 16:07:49.499 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: log_rotate = 4
2024-04-17 16:07:49.500 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: log_size = 1048576
2024-04-17 16:07:49.501 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: logdir = /var/log
2024-04-17 16:07:49.502 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: logfilelevel = 9
2024-04-17 16:07:49.503 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: max_parallel_downloads = 3
2024-04-17 16:07:49.505 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: metadata_expire = 172800
2024-04-17 16:07:49.506 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: metadata_timer_sync = 10800
2024-04-17 16:07:49.507 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: minrate = 1000
2024-04-17 16:07:49.508 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: module_obsoletes = 0
2024-04-17 16:07:49.509 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: module_stream_switch = 0
2024-04-17 16:07:49.510 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: multilib_policy = best
2024-04-17 16:07:49.511 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: obsoletes = 1
2024-04-17 16:07:49.512 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: password = 
2024-04-17 16:07:49.513 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: persistdir = /var/lib/dnf
2024-04-17 16:07:49.514 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: pluginconfpath = /etc/dnf/p
lugins
2024-04-17 16:07:49.515 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: pluginpath = /usr/lib/pytho
n3.6/site-packages/dnf-plugins
2024-04-17 16:07:49.516 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: plugins = 1
2024-04-17 16:07:49.518 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: protect_running_kernel = 1
2024-04-17 16:07:49.519 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: protected_packages = dnf, s
etup, systemd, systemd-udev, dnf
2024-04-17 16:07:49.520 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy = 
2024-04-17 16:07:49.521 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy_auth_method = any
2024-04-17 16:07:49.522 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy_sslcacert = 
2024-04-17 16:07:49.523 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy_sslclientcert = 
2024-04-17 16:07:49.524 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy_sslclientkey = 
2024-04-17 16:07:49.525 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: proxy_sslverify = 1
2024-04-17 16:07:49.526 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: recent = 7
2024-04-17 16:07:49.528 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: repo_gpgcheck = 0
2024-04-17 16:07:49.529 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: reposdir = /etc/yum.repos.d
, /etc/yum/repos.d, /etc/distro.repos.d
2024-04-17 16:07:49.530 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: reset_nice = 1
2024-04-17 16:07:49.531 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: retries = 10
2024-04-17 16:07:49.532 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: rpmverbosity = info
2024-04-17 16:07:49.533 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: showdupesfromrepos = 0
2024-04-17 16:07:49.534 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: skip_broken = 0
2024-04-17 16:07:49.535 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: skip_if_unavailable = 0
2024-04-17 16:07:49.536 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: sslcacert = 
2024-04-17 16:07:49.537 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: sslclientcert = 
2024-04-17 16:07:49.538 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: sslclientkey = 
2024-04-17 16:07:49.540 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: sslverify = 1
2024-04-17 16:07:49.541 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: sslverifystatus = 0
2024-04-17 16:07:49.542 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: strict = 1
2024-04-17 16:07:49.543 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: system_cachedir = /var/cach
e/dnf
2024-04-17 16:07:49.544 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: throttle = 0
2024-04-17 16:07:49.545 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: timeout = 30
2024-04-17 16:07:49.546 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: transformdb = 1
2024-04-17 16:07:49.547 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: tsflags = 
2024-04-17 16:07:49.549 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: upgrade_group_objects_upgra
de = 1
2024-04-17 16:07:49.550 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: user_agent = libdnf (Rocky 
Linux 8.9; generic; Linux.x86_64)
2024-04-17 16:07:49.550 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: username = 
2024-04-17 16:07:49.551 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: varsdir = /etc/yum/vars, /e
tc/dnf/vars
2024-04-17 16:07:49.553 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: zchunk = 1
2024-04-17 16:07:49.554 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: 
2024-04-17 16:07:49.556 DEBUG    PID: 21742 leapp.workflow.Download.dnf_package_download: External command has finish
ed: ['systemd-nspawn', '--register=no', '--quiet', '-D', u'/var/lib/leapp/el8userspace', '--setenv=LEAPP_COMMON_FILES
=:/etc/leapp/repos.d/system_upgrade/wp-toolkit/files:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.
d/system_upgrade/el7toel8/files', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=8.6', '--setenv=LEAPP_NO_RHSM=1', '--se
tenv=LEAPP_EXPERIMENTAL=0', '--sete
nv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/r
epos.d/system_upgrade/cloudlinux/tools:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgra
de/el7toel8/tools', '--setenv=LEAPP_NOGPGCHECK=1', '--setenv=LEAPP_IPU_IN_PROGRESS=7to8', '--setenv=LEAPP_UNSUPPORTED
=0', '--setenv=LEAPP_EXECUTION_ID=0ba998a0-72ed-4eed-9bd5-2fa106e678d6', '--setenv=LEAPP_HOSTNAME=qtp-r8-mig-tony.c.f
errum-dev.internal', 'dnf', 'config-manager', '--dump']
pirat89 commented 6 months ago

Hi, it's known limitation that's been fixed in version 0.20.0:

The installed version (0.16.0) has been released 2y ago.

verdverm commented 6 months ago

@pirat89 I grabbed the latest master...el7 packages from: https://copr.fedorainfracloud.org/coprs/g/oamg/leapp/packages/

Now I am seeing: Error: Unexpected format of target version: 7 when running leapp preupgrade

Any ideas on what this issue is?

verdverm commented 6 months ago

Seems this is the issue: https://github.com/oamg/leapp-repository/issues/1126

Further complicating the issue is that elevate is not updating the leapp packages, stuck on old versions, so catch 22 for

pirat89 commented 6 months ago

yes, that's it.

verdverm commented 6 months ago

For those who come along later, wanting to upgrade Centos7 -> Rocky8 with a local repo mirror

python3 -m http.server 1337 --bind 127.0.0.1 works around the issue, and leapp/dnf see it as a remote repository

We hit an issue where we build a single, minimal yum repo from all the packages we need. This hit another error for a number of packages like...

No available modular metadata for modular package 'fuse-overlayfs-1.12-1.module+el8.9.0+1703+29de406e.x86_64', it cannot be installed on the system

It seems we have to

  1. match the layout for the packages / multi-repo setup
  2. use createrepo_c instead of createrepo for each

https://forums.rockylinux.org/t/fix-no-available-modular-metadata-package-offline-repo/8713

FWIW, we are going to discuss/push for access to the cloud mirrors given how complicated this has become