error: importing RPMs: digest: SIGNATURE: NOT OK (was: SHA-1 signed RPMs (Chrome) on Silverblue 38)

[travier]

Updated issue text

See: https://github.com/fedora-silverblue/issue-tracker/issues/408#issuecomment-1570733586

error: importing RPMs: package google-chrome-stable-114.0.5735.90-1.x86_64 cannot be verified and repo google-chrome is GPG enabled: /var/cache/rpm-ostree/repomd/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm could not be verified.
/var/cache/rpm-ostree/repomd/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm:  digest:  SIGNATURE:  NOT OK

See also: https://discussion.fedoraproject.org/t/google-chrome-failed-update-due-to-signature-not-ok/83540

---

Original issue text

Describe the bug

SHA-1 signed RPMs will likely trigger failures on Fedora Silverblue 38.

See: https://www.scrye.com/wordpress/nirik/2023/01/31/error-rpmdbnextiterator-skipping-in-fedora-38/

To Reproduce

Please describe the steps needed to reproduce the bug:

1. Install the Chrome RPM on Fedora 38 with rpm-ostree install

--

Needs to be investigated to see if the workaround from https://www.scrye.com/wordpress/nirik/2023/01/31/error-rpmdbnextiterator-skipping-in-fedora-38/ works with rpm-ostree as well or if we need something else.

[tpopela]

This is currently tracked in https://bugs.chromium.org/p/chromium/issues/detail?id=1383526 . Maybe it will be resolved upstream in the meantime (or not).

[travier]

Looks like I can not reproduce this issue on F38.

[travier]

Maybe something changed here as I now get:

error: importing RPMs: package google-chrome-stable-114.0.5735.90-1.x86_64 cannot be verified and repo google-chrome is GPG enabled: /var/cache/rpm-ostree/repomd/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm could not be verified.
/var/cache/rpm-ostree/repomd/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm:  digest:  SIGNATURE:  NOT OK

See also: https://discussion.fedoraproject.org/t/google-chrome-failed-update-due-to-signature-not-ok/83540

[ghost]

It isn't a Silverblue specific issue. This also happens in the Workstation edition (38).

$ pkcon update
Getting updates               [=========================]         
Finished                      [=========================]         
Starting                      [=========================]         
Testing changes               [=========================]         
Finished                      [=========================]         
Updating packages             [=========================]         
Querying                      [=========================]         
Finished                      [=========================]         rinter drivers
Fatal error: package google-chrome-stable-114.0.5735.90-1.x86_64 cannot be verified and repo google-chrome is GPG enabled: /var/cache/PackageKit/38/metadata/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm could not be verified.? [N/y] y
/var/cache/PackageKit/38/metadata/google-chrome-38-x86_64/packages/google-chrome-stable-114.0.5735.90-1.x86_64.rpm:  digest:  SIGNATURE:  NOT OK

[tpopela]

Yes, this is not related to Silverblue at all. We've seen this in our RHEL CI last week as well. What happened is that Chromium rotated their signing keys, see https://chromium.googlesource.com/chromium/src/+/a09e77359c939913f6a84ae43ed81bea32f9b46e%5E%21/#F1 and https://chromium-review.googlesource.com/c/chromium/src/+/4404987, but I suspect that something went wrong there..

[ghost]

Yes, this is not related to Silverblue at all. We've seen this in our RHEL CI last week as well. What happened is that Chromium rotated their signing keys, see https://chromium.googlesource.com/chromium/src/+/a09e77359c939913f6a84ae43ed81bea32f9b46e%5E%21/#F1 and https://chromium-review.googlesource.com/c/chromium/src/+/4404987, but I suspect that something went wrong there..

I think it's better to file a report in bugs.chromium.org. However, I don't want to use my personal account. D:

[travier]

Re-installing from the latest RPM might work (not tried yet):

$ rpm-otree update --uninstall google-chrome-stable
$ reboot
$ rpm-ostree install google-chrome-stable.rpm
$ reboot
$ rpm-ostree update --uninstall google-chrome-stable-<version> --install google-chrome-stable
$ reboot

[heyakyra]

this is not related to Silverblue at all

But for Silverblue, is there a way to exclude a single package or repo from an upgrade command?

[bnordgren]

People will probably emit gasps of horror at this, but you can always edit /etc/yum.repos.d/google-chrome.repo and set "gpgcheck=0". Then try again.

Signing a package isn't magic. All it means is that the package is the same as when it was signed. In this case, since the public key and the package are provided by exactly the same website, the same entity is controlling both. If it's google, yippie. If it's a competent hacker who replaced both the package and the signing key, you'll never know. The only thing you're going to catch by comparing signatures is: 1] upgrade-preventing (but otherwise harmless) mistakes by the official release crew; and 2] incompetent hackers who replace either the package or the signing key but not both.

Since we have confirmation through other channels that the answer is in fact behind door number 1, I don't feel that disabling the check places us in a substantially worse position than being unable to distinguish whether a successful signature verification was legitimate or a false positive.

[616b2f]

Re-installing from the latest RPM might work (not tried yet):

$ rpm-otree update --uninstall google-chrome-stable
$ reboot
$ rpm-ostree install google-chrome-stable.rpm
$ reboot
$ rpm-ostree update --uninstall google-chrome-stable-<version> --install google-chrome-stable
$ reboot

This worked for me, but I only did the rpm-ostree install google-chrome-stable.rpm step. The RPM downloaded from the official Google Chrome website.

[travier]

Re-installing from the latest RPM might work (not tried yet):

$ rpm-otree update --uninstall google-chrome-stable
$ reboot
$ rpm-ostree install google-chrome-stable.rpm
$ reboot
$ rpm-ostree update --uninstall google-chrome-stable-<version> --install google-chrome-stable
$ reboot

This worked for me, but I only did the rpm-ostree install google-chrome-stable.rpm step. The RPM downloaded from the official Google Chrome website.

If you don't re-install it at the end, you'll be stuck with a fixed version of the Google Chrome package. See https://discussion.fedoraproject.org/t/simplifying-updates-for-rpm-fusion-packages-and-other-packages-shipping-their-own-rpm-repos/30364

[travier]

Another untested option would be to fetch the new public key used and replace the old one in the config and then update, which should come with the repo config and new signing key.

I'll close this issue as while this only happens on rpm-ostree based systems, there is not much we can do to fix it.

[tmstaedt]

Just had this problem, too, trying to update fedora 38 on silverblue. Have chrome dev version installed As a workaround, uninstalled the rpm-ostree version, installed the flatpak version via gnome-software. Looks good so far.

[luminoso]

Just had this problem, too, trying to update fedora 38 on silverblue. Have chrome dev version installed As a workaround, uninstalled the rpm-ostree version, installed the flatpak version via gnome-software. Looks good so far.

If it is this one: https://flathub.org/apps/com.google.Chrome

Two problems with that:

1. Flatpak version is still 113.x version, not 114.x that has this issue
2. The wrapper is not verified by, affiliated with, or supported by Google

[tmstaedt]

Just had this problem, too, trying to update fedora 38 on silverblue. Have chrome dev version installed As a workaround, uninstalled the rpm-ostree version, installed the flatpak version via gnome-software. Looks good so far.

If it is this one: https://flathub.org/apps/com.google.Chrome

Two problems with that:

1. Flatpak version is still 113.x version, not 114.x that has this issue

But Chrome dev Version is 116, updated a few days ago. I use it almost since I use Silverblue, and never had issues with it. Regular version flatpak probably follows soon.

2. The wrapper is not verified by, affiliated with, or supported by Google

why should it? Google maintains the browser.

[kees-closed]

why should it? Google maintains the browser.

But how can you know with certainty that the software is not modified? With RPMs there are some standardized checks and balances based on hashes and possibly signed tar files. Flatpaks that are not from the developer itself are just the wild west. Some are .deb files, some are Bash scripts extracting files. None is really standardized and automatically checked for inconsistencies.

[616b2f]

Re-installing from the latest RPM might work (not tried yet):

$ rpm-otree update --uninstall google-chrome-stable
$ reboot
$ rpm-ostree install google-chrome-stable.rpm
$ reboot
$ rpm-ostree update --uninstall google-chrome-stable-<version> --install google-chrome-stable
$ reboot

This worked for me, but I only did the rpm-ostree install google-chrome-stable.rpm step. The RPM downloaded from the official Google Chrome website.

If you don't re-install it at the end, you'll be stuck with a fixed version of the Google Chrome package. See https://discussion.fedoraproject.org/t/simplifying-updates-for-rpm-fusion-packages-and-other-packages-shipping-their-own-rpm-repos/30364

I was to fast, while the installing the RPM works, as you said to be able to update I did the uninstall RPM package step and trying to install it via repository, which does not work. It fails with the same error.

[tmstaedt]

why should it? Google maintains the browser.

But how can you know with certainty that the software is not modified? With RPMs there are some standardized checks and balances based on hashes and possibly signed tar files. Flatpaks that are not from the developer itself are just the wild west. Some are .deb files, some are Bash scripts extracting files. None is really standardized and automatically checked for inconsistencies.

Naaa. The flatpaks, which ship 3rd party binaries, are also key-checked from the originating upstream download site. No wild west here. Correct me if I'm wrong.

[increasingawareness]

A few weeks out and still facing this issue. I see this was filed with the Chromium team but can't find anything else. Still can't tell if someone at Google messed up the key signing or they are changing the way they are signing and rpm-ostree can't handle it.

[travier]

Re-installing from the latest RPM might work (not tried yet):

$ rpm-otree update --uninstall google-chrome-stable
$ reboot
$ rpm-ostree install google-chrome-stable.rpm
$ reboot
$ rpm-ostree update --uninstall google-chrome-stable-<version> --install google-chrome-stable
$ reboot

So this does not work as the new key (https://chromium.googlesource.com/chromium/src/+/main/chrome/installer/linux/common/rpm.include) is not available at https://dl.google.com/linux/linux_signing_key.pub which is what's used by the repo (https://chromium.googlesource.com/chromium/src/+/main/chrome/installer/linux/common/rpm.include#359):

[google-chrome]
name=google-chrome
baseurl=https://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub

Someone should ask them to update the list of key published at https://dl.google.com/linux/linux_signing_key.pub.

I'm out of ideas for workarounds. I've moved to the Flatpak despite its problems (partial FIDO device support).

[travier]

We've found another potential fix:

• Add the full set of GPG keys to https://github.com/xsuchy/distribution-gpg-keys/tree/main/keys/google
• Update https://src.fedoraproject.org/rpms/fedora-workstation-repositories/blob/rawhide/f/google-chrome.repo to use those keys instead
• Test if this works
• Write docs to help users "reset" their config and update their systems

[pbevilac]

For me sudo dnf upgrade --refresh updated Chrome OK, by accepting the keys warnings. Chrome itself was warning about inability to self-update. Fedora remained to 36 however.

I think that the piece of GUI named 'software' should do better in handling this type of problem, where a third-party failure can completely block a system update from GUI, that just showing (non-copyable) error text. Not all users are able to quickly switch to CLI for routine updates.

[travier]

This is the issue tracker for Fedora Silverblue. Please file this issue in the GNOME Software tracker.

[increasingawareness]

I rebased today to Rawhide to check if this issue is resolved and I'm still running into it. From what I can tell, RPM does not support signing of packages with subkeys, which newer releases of Chrome are signed with by the team. The fix would be be for RPM (or RPM-ostree in this case) to add subkeys to supported signing methods, no?

Edit: To be clear, I can create a toolbox and install chrome just fine with dnf (install RPM file, remove numbered package and install from repo by sudo dnf install google-chrome-stable). It prompts to accept or reject the keys from Google along with the subkey signing key.

Edit2: I can also follow the instructions on https://google.com/linuxrepositories and verify the key in toolbox, but cannot in a normal silverblue terminal because it's not writable (of course being immutable). I wonder if this is due to something in RPM-ostree specifically, as the output in toolbox (RPM) adds the subkeys appropriately.

Edit3: The toolbox version doesn't support FIDO2 login, but I was able to use webauthn on Yubico's website, so idk what's happening there. But anyway, this appears to me not to be an issue upstream with Google, as the subkey is there. The problem seems to be how RPM-ostree handles things signed with subkeys. Hopefully this helps people also looking at this issue.

[travier]

https://discussion.fedoraproject.org/t/google-chrome-failed-update-due-to-signature-not-ok/83540/38

[Malix-off]

I successfully installed google-chrome-stable by removing gpgcheck

1. nano /etc/yum.repos.d/google-chrome.repo
2. Set gpgcheck to 0 (gpgcheck=0)
3. Save (^X + Y + Enter)
4. Replace local layer with repo layer: rpm-ostree uninstall google-chrome-stable && rpm-ostree install google-chrome-stable

I know this is not the best option, but it worked. I don't know if I will receive updates too, but I think I will; It will just not check for the gpg key, which is a bad practice.

If someone know how I can restore the gpg key check, I will appreciate any help

[Malix-off]

@increasingawareness

RPM does not support signing of packages with subkeys, which newer releases of Chrome are signed with by the team. The fix would be be for RPM (or RPM-ostree in this case) to add subkeys to supported signing methods, no?

I agree
Would you mind creating the chromium issue and linking it here ?

[tpopela]

This is also tracked in https://github.com/rpm-software-management/dnf/issues/2080 and https://bugzilla.redhat.com/show_bug.cgi?id=2274169