search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
264 stars 59 forks source link

rawhide: 20210814: rpm-ostree install $pkg failing #925

Closed dustymabe closed 3 years ago

dustymabe commented 3 years ago

Rawhide builds based on the 20210814 rpm package set starting failing over the weekend. link

[core@cosa-devsh ~]$ sudo rpm-ostree install bird
Checking out tree 0777f1c... done
Enabled rpm-md repositories: fedora-cisco-openh264 rawhide
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'rawhide'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2021-02-23T00:47:28Z solvables: 4
rpm-md repo 'rawhide'; generated: 2021-08-14T08:29:12Z solvables: 65430
Resolving dependencies... done
Will download: 1 package (437.5?kB)
Downloading from 'rawhide'... done
error: PKI file /var/cache/rpm-ostree/repomd/rawhide-rawhide-x86_64/RPM-GPG-KEY-fedora-37-$basearch is not a public key

Not really sure why it's trying to use the Fedora 37 key instead of 36 but the key for x86_64 does seem to be a public key:

[core@cosa-devsh ~]$ file -L /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-37-x86_64                                                                                                                                                                                                                                                                         
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-37-x86_64: PGP public key block
dustymabe commented 3 years ago

Here's a clue in /etc/yum.repos.d/rawhide.repo:

[rawhide]
name=Fedora - Rawhide - Developmental packages for the next Fedora release
#baseurl=http://download.example/pub/fedora/linux/development/rawhide//Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch
enabled=1
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-$basearch file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-37-$basearch
skip_if_unavailable=False
dustymabe commented 3 years ago

I guess the 37 in there is expected according to the spec file.

Next thing I'm looking at:

[vagrant@vanilla-rawhide rpm-gpg]$ file -L RPM-GPG-KEY-fedora-36-primary
RPM-GPG-KEY-fedora-36-primary: PGP public key block Public-Key (old)
[vagrant@vanilla-rawhide rpm-gpg]$ file -L RPM-GPG-KEY-fedora-37-primary
RPM-GPG-KEY-fedora-37-primary: PGP public key block

Maybe rpm-ostree can't handle the new file format that's being used?

cgwalters commented 3 years ago

This is probably that libdnf doesn't understand that space separated syntax, I bet one would hit the same issue with microdnf. repo file parsing is still split between dnf/libdnf.

cgwalters commented 3 years ago

Er, it may be libdnf isn't substituting $basearch there.

dustymabe commented 3 years ago

Er, it may be libdnf isn't substituting $basearch there.

Set of package diff since last successful run:

 Upgraded:
   NetworkManager 1:1.32.6-1.fc35 -> 1:1.32.8-1.fc35
   NetworkManager-cloud-setup 1:1.32.6-1.fc35 -> 1:1.32.8-1.fc35
   NetworkManager-libnm 1:1.32.6-1.fc35 -> 1:1.32.8-1.fc35
   NetworkManager-team 1:1.32.6-1.fc35 -> 1:1.32.8-1.fc35
   NetworkManager-tui 1:1.32.6-1.fc35 -> 1:1.32.8-1.fc35
   afterburn 5.0.0-2.fc35 -> 5.1.0-1.fc36
   afterburn-dracut 5.0.0-2.fc35 -> 5.1.0-1.fc36
   audit-libs 3.0.3-2.fc35 -> 3.0.5-1.fc36
   chrony 4.1-2.fc35 -> 4.1-3.fc35
   container-selinux 2:2.164.2-1.fc35 -> 2:2.164.2-1.fc36
   containers-common 4:1-24.fc35 -> 4:1-25.fc36
   crun 0.21-1.fc35 -> 0.21-1.fc36
   fedora-gpg-keys 35-0.4 -> 36-0.2
   fedora-release-common 35-0.14 -> 36-0.2
   fedora-release-coreos 35-0.14 -> 36-0.2
   fedora-release-identity-coreos 35-0.14 -> 36-0.2
   fedora-repos 35-0.4 -> 36-0.2
   fedora-repos-archive 35-0.4 -> 36-0.2
   fedora-repos-modular 35-0.4 -> 36-0.2
   fedora-repos-ostree 35-0.4 -> 36-0.2
   fedora-repos-rawhide 35-0.4 -> 36-0.2
   fedora-repos-rawhide-modular 35-0.4 -> 36-0.2
   firewalld-filesystem 1.0.0-1.fc35 -> 1.0.1-1.fc36
   fuse-overlayfs 1.7.0-1.fc35 -> 1.7.1-1.fc35
   fwupd 1.6.2-1.fc35 -> 1.6.3-1.fc35
   iscsi-initiator-utils 6.2.1.4-0.git095f59c.fc35.2 -> 6.2.1.4-1.git2a8f9d8.fc36
   iscsi-initiator-utils-iscsiuio 6.2.1.4-0.git095f59c.fc35.2 -> 6.2.1.4-1.git2a8f9d8.fc36
   jq 1.6-7.fc34 -> 1.6-9.fc36
   kernel 5.14.0-0.rc4.20210804gitd5ad8ec3cfb5.36.fc35 -> 5.14.0-0.rc5.20210811git761c6d7ec820.44.fc36
   kernel-core 5.14.0-0.rc4.20210804gitd5ad8ec3cfb5.36.fc35 -> 5.14.0-0.rc5.20210811git761c6d7ec820.44.fc36
   kernel-modules 5.14.0-0.rc4.20210804gitd5ad8ec3cfb5.36.fc35 -> 5.14.0-0.rc5.20210811git761c6d7ec820.44.fc36
   kmod 29-4.fc35 -> 29-5.fc36
   kmod-libs 29-4.fc35 -> 29-5.fc36
   less 581.2-2.fc35 -> 590-1.fc35
   libipa_hbac 2.5.2-3.fc35 -> 2.5.2-4.fc35
   libpwquality 1.4.4-5.fc35 -> 1.4.4-6.fc36
   libsmbclient 2:4.15.0-0.2.rc1.fc35 -> 2:4.15.0-0.4.rc2.fc36
   libsss_certmap 2.5.2-3.fc35 -> 2.5.2-4.fc35
   libsss_idmap 2.5.2-3.fc35 -> 2.5.2-4.fc35
   libsss_nss_idmap 2.5.2-3.fc35 -> 2.5.2-4.fc35
   libsss_sudo 2.5.2-3.fc35 -> 2.5.2-4.fc35
   libwbclient 2:4.15.0-0.2.rc1.fc35 -> 2:4.15.0-0.4.rc2.fc36
   libxcrypt 4.4.24-1.fc35 -> 4.4.25-1.fc36
   mdadm 4.1-9.fc35 -> 4.2-rc2.fc35
   mozjs78 78.12.0-3.fc35 -> 78.13.0-1.fc35
   samba-client-libs 2:4.15.0-0.2.rc1.fc35 -> 2:4.15.0-0.4.rc2.fc36
   samba-common 2:4.15.0-0.2.rc1.fc35 -> 2:4.15.0-0.4.rc2.fc36
   samba-common-libs 2:4.15.0-0.2.rc1.fc35 -> 2:4.15.0-0.4.rc2.fc36
   selinux-policy 34.15-1.fc35 -> 34.16-1.fc36
   selinux-policy-targeted 34.15-1.fc35 -> 34.16-1.fc36
   skopeo 1:1.4.0-1.fc35 -> 1:1.4.0-1.fc36
   sssd-ad 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-client 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-common 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-common-pac 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-ipa 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-krb5 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-krb5-common 2.5.2-3.fc35 -> 2.5.2-4.fc35
   sssd-ldap 2.5.2-3.fc35 -> 2.5.2-4.fc35
   toolbox 0.0.99.2^2.git40fbd377ed0b-1.fc35 -> 0.0.99.2^3.git075b9a8d2779-1.fc35
   vim-minimal 2:8.2.3290-1.fc35 -> 2:8.2.3318-1.fc35

DNF didn't get an update so probably not that?

dustymabe commented 3 years ago

More info.. microdnf barfs:

bash-5.1# microdnf install bird 
Downloading metadata...
Downloading metadata...
Package                                                                                           Repository                             Size
Installing:                                                                                                                                  
 bird-2.0.8-2.fc35.x86_64                                                                         rawhide                     437.5\xc2\xa0kB
 libsemanage-3.2-4.fc35.x86_64                                                                    rawhide                     118.3\xc2\xa0kB
 shadow-utils-2:4.9-2.fc36.x86_64                                                                 rawhide                       1.2\xc2\xa0MB
Transaction Summary:
 Installing:        3 packages
 Reinstalling:      0 packages
 Upgrading:         0 packages
 Obsoleting:        0 packages
 Removing:          0 packages
 Downgrading:       0 packages
Is this ok [y/N]: y
Downloading packages...

(microdnf:3): libdnf-WARNING **: 19:56:51.924: PKI file /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-37-primary is not a public key
error: PKI file /var/cache/yum/metadata/rawhide-rawhide-x86_64/RPM-GPG-KEY-fedora-37-$basearch is not a public key
cgwalters commented 3 years ago

I bet this broke from https://src.fedoraproject.org/rpms/fedora-repos/c/1db958c4fbd16ec2000db953424aa0317e19b125?branch=rawhide

dustymabe commented 3 years ago

I thought so too, but that change has been in place for 6 months. I don't think it's the problem here. I'm pretty sure it's the file format change of the new key (introduced in https://src.fedoraproject.org/rpms/fedora-repos/c/688de4b2d4bf3411433d7e0978030b3a2cb37795?branch=rawhide)

cgwalters commented 3 years ago

Ahhh wow, actually it looks like dnf uses gpg/gpgme in https://github.com/rpm-software-management/dnf/blob/395541fbf8f87f81cdca7567f22be1182e55bea7/dnf/crypto.py but libdnf uses rpm's bespoke PGP code.

(ostree uses gpgme)

dustymabe commented 3 years ago

https://github.com/rpm-software-management/libdnf/issues/1320

dustymabe commented 3 years ago

PR to workaround issue for now so we can keep rawhide building: https://github.com/coreos/fedora-coreos-config/pull/1163

dustymabe commented 3 years ago

going to close this since https://github.com/coreos/fedora-coreos-config/commit/e160b580869815d3c94aae531ab9cfac93239b2b landed.