search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
262 stars 59 forks source link

[stable][aarch64] ext.config.extensions.package kola test failure #1776

Open c4rt0 opened 1 month ago

c4rt0 commented 1 month ago

Describe the bug

During fcos (40.20240728.3.0) release [stable][aarch64] ext.config.extensions.package kola test failed. I was able to reproduce this in a debug pod, confirming that test passes with previous stable release.

Reproduction steps

  1. Run [stable][aarch64] debug pod
  2. verify VERSION="40.20240814.dev.0 (CoreOS)":
  3. Run rpm-ostree install --apply-live vim
  4. Observe failure
  5. Downgrade to 40.20240709.dev.0 with sudo rpm-ostree upgrade 40.20240709.3.1 --allow-downgrade
  6. Succesfully Run rpm-ostree install --apply-live vim

Expected behavior

Test passes. On the debug pod vim is installed as expected on latest [stable[aarch64] build.

Actual behavior

Test fails.

on the debug-pod installation is not successful. log:

[core@cosa-devsh ~]$ sudo rpm-ostree install vim
error: importing RPMs: package vim-common-2:9.1.571-1.fc40.aarch64 cannot be verified and repo updates-archive is GPG enabled: /var/cache/rpm-ostree/repomd/updates-archive-40-aarch64/packages/vim-common-9.1.571-1.fc40.aarch64.rpm could not be verified.
/var/cache/rpm-ostree/repomd/updates-archive-40-aarch64/packages/vim-common-9.1.571-1.fc40.aarch64.rpm:  DIGEST:  SIGNATURE:  NOT OK
[core@cosa-devsh ~]$ sudo rpm-ostree install vim
Checking out tree 5e92204... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2024-03-12T11:45:42Z solvables: 3
rpm-md repo 'updates' (cached); generated: 2024-08-14T06:21:14Z solvables: 21034
rpm-md repo 'fedora' (cached); generated: 2024-04-14T18:51:04Z solvables: 65497
rpm-md repo 'updates-archive' (cached); generated: 2024-08-09T03:46:38Z solvables: 31831
Resolving dependencies... done
Will download: 3 packages (191.8_kB)
Downloading from 'updates'... done
Importing packages... done
error: importing RPMs: package vim-common-2:9.1.571-1.fc40.aarch64 cannot be verified and repo updates-archive is GPG enabled: /var/cache/rpm-ostree/repomd/updates-archive-40-aarch64/packages/vim-common-9.1.571-1.fc40.aarch64.rpm could not be verified.
/var/cache/rpm-ostree/repomd/updates-archive-40-aarch64/packages/vim-common-9.1.571-1.fc40.aarch64.rpm:  DIGEST:  SIGNATURE:  NOT OK

System details

[core@cosa-devsh ~]$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="40.20240814.dev.0 (CoreOS)"
ID=fedora
VERSION_ID=40
VERSION_CODENAME=""
PLATFORM_ID="platform:f40"
PRETTY_NAME="Fedora CoreOS 40.20240814.dev.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:40"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=40
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=40
SUPPORT_END=2025-05-13
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='40.20

Butane or Ignition config

No response

Additional information

No response

jlebon commented 1 month ago

Likely the archive-repo-manager got interrupted in the past on that point and the RPM is incomplete. I think we just need to delete the RPM there and the code will know to refetch it.

dustymabe commented 1 month ago

Likely the archive-repo-manager got interrupted in the past on that point and the RPM is incomplete. I think we just need to delete the RPM there and the code will know to refetch it.

Close..

  1. delete the existing RPM files in S3 that we think are corrupt
  2. re-download them using koji download-build

Unfortunately the archive repo manager won't pick up that it was deleted and re-download it. The way it measures progress is based on a timestamp file and the only way to make it re-download a file is if you were to reset the timestamp file back to an earlier timestamp.

I guess we can try to enhance the ARM to verify the files when they are first downloaded and retry if they somehow are incomplete.