search

holland-backup / holland

Holland Backup Manager
http://hollandbackup.org
Other
152 stars 49 forks source link

OBS release key expired 2023-03-06 #360

Closed terackspace closed 1 year ago

terackspace commented 1 year ago

@soulen3 - I was alerted this key expired yesterday:

I see there's a new GPG key that's set up for expiration in 2025 over here (using my project login):

...so I think we need to just resign all the repos so the updated key is deployed to each versioned subdirectory? I don't think my login has access to the private keypart so maybe it's an option only you can see in their webUI? Thanks!

soulen3 commented 1 year ago

Ubuntu 18.04 looks like it's going EOL soon. So I've removed it from the repo list and added 22.10

mwsessions commented 1 year ago

This also appears to affect:

Ubuntu 20.04 Ubuntu 21.04 Ubuntu 21.10 Ubuntu 22.04 Debian 9-11 Raspbian 9, 11

soulen3 commented 1 year ago

I'm not sure what the issues is then. But @terackspace has the same level of access as me.

terackspace commented 1 year ago

I don't have (or, if I do I can't see/find it) access to the private keypart which I think is needed; I'm just searching around here but it appears that maybe one hast to use this osc tool with the private key in their hands to update it? https://forums.opensuse.org/t/how-to-update-expired-obs-home-project-signing-key/119147 @abg you have any idea how this works to re-sign the repos with an updated key?

terackspace commented 1 year ago

(working notes)

My home:terackspace GPG key had also expired (never used), so I followed the above link to test that action:

$ sudo apt-get install osc
$ osc --no-keyring signkey --extend home:terackspace
 Your user account / password are not configured yet.
   (username, password mumbo jumbo)
   7) Transient password store (Do not store the password and always ask for the password)
   Select credentials manager: 7
 done
 Password: 
 <status code="ok" />

And this did update the project GPG signing key in my test home project, without needing any sort of private keypart (which I don't think I have for this). It's still unclear how I would propagate this to existing repos, there is a release subcommand of this osc tool but it doesn't directly have a "resign repos" command that I can suss out.

The GPG located here: https://build.opensuse.org/project/keys_and_certificates/home:holland-backup has an updated expiration date of 2025-04-30 already -- it might fix our problem if we made a new release of the Holland package (just bump the release number, maybe?) which would force all the repos to get updated and resigned. Just a guess though...

terackspace commented 1 year ago

(working notes)

I got Paul into the project, we started poking around and just tried this test which seems to have worked (not failed); it's heavily cached on mirrors so your specific location might not be updated yet..

$ osc rebuildpac home:holland-backup holland.deb Debian_8.0

...now has the updated Release.key with the 2025 date on it. It looks like we could trigger an update to all repos (some are already up to date, like Raspbian_10 but it shouldn't matter) without specifying the repo, just:

$ osc rebuildpac home:holland-backup holland.deb

This is most likely our next step, unless a better idea comes up here on how to tackle the problem.

terackspace commented 1 year ago

Since there's no other feedback on a better way to tackle this, I'll go ahead and run the command to rebuild all packages to get the new GPG key used tomorrow morning (Tuesday, 14 March, CDT). Holler now if you don't like the plan.

terackspace commented 1 year ago 
$ date
Tue Mar 14 08:00:13 AM CDT 2023

$ osc rebuildpac home:holland-backup holland.deb
Password: 
ok

The GUI at https://build.opensuse.org/project/show/home:holland-backup is now showing all packages in Rebuild state...

terackspace commented 1 year ago

All the builds appear to have succeeded and are live to go out to the mirrors; because it was trivial to do, we re-enabled Ubuntu 18.04 since it's technically not EOL yet and doesn't hurt to have around for those folks who need it. I think we're all done here, I'll close up the issue.

Sort by Last Modified: https://download.opensuse.org/repositories/home:/holland-backup/

tmrsiteops commented 1 year ago

Hi, I was getting the expired key message when I attempted to install a few days ago. That has cleared up, but now when I run apt-get install holland I see this output:

The following NEW packages will be installed:
  holland
0 upgraded, 1 newly installed, 0 to remove and 57 not upgraded.
Need to get 96.7 kB of archives.
After this operation, 562 kB of additional disk space will be used.
Get:1 https://download.opensuse.org/repositories/home:/holland-backup/Debian_10 ./ holland 1.2.10-0 [96.7 kB]
Err:1 https://download.opensuse.org/repositories/home:/holland-backup/Debian_10 ./ holland 1.2.10-0
  File has unexpected size (96740 != 96712). Mirror sync in progress? [IP: 72.4.120.219 443]
  Hashes of expected file:
   - SHA256:1f825c5d08d857f4ee9cc1a51f4cce61625ddd673b1230d42cd4bf020fd4b118
   - SHA1:70d5eeed25273dca12aa54cd8853ae8dc202d4e8 [weak]
   - MD5Sum:632a034ca5c2275b7808574465dc8971 [weak]
   - Filesize:96712 [weak]
E: Failed to fetch https://mirror.rackspace.com/openSUSE/repositories/home:/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb  File has unexpected size (96740 != 96712). Mirror sync in progress? [IP: 72.4.120.219 443]
   Hashes of expected file:
    - SHA256:1f825c5d08d857f4ee9cc1a51f4cce61625ddd673b1230d42cd4bf020fd4b118
    - SHA1:70d5eeed25273dca12aa54cd8853ae8dc202d4e8 [weak]
    - MD5Sum:632a034ca5c2275b7808574465dc8971 [weak]
    - Filesize:96712 [weak]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

This is in Debian 10

terackspace commented 1 year ago

I think I see what you see, using these two files directly:

They don't match up:

$ sha256sum holland_1.2.10-0_all.deb 
0b93750c21fbed005105689dcb1767427e1b15787fce1d0566de63fe51456645  holland_1.2.10-0_all.deb

$ grep SHA256 Packages
SHA256: 1f825c5d08d857f4ee9cc1a51f4cce61625ddd673b1230d42cd4bf020fd4b118

This is all supposed to be OBS magic, but it appears to not have updated the repo metadata when rebuilding the packages. We'll have to dig around these tools a bit and see what's to be done to make it work as expected.

terackspace commented 1 year ago

After an hour of searching around the internet, reading every doc I can find and looking at every tooling command - this sure feels like a bug and I'm unsure the best way to proceed. Every other index file is rebuilt (for example InRelease and Release) to have updated checksums and GPG key, but the Release file is not updated for the newly build package checksums.

The only possible thing I see is a "wipebinaries" command ("unpublish") - there's no publish, republish and the "release" feature requires the project to be set to fully manual instead of automatic (so that's off the table). But then it's not clear, if we wipe the binaries do we... re-run rebuildpac to recreate them? Just like there's no "just resign the repos with a new GPG key", we seem to lack a "just republish a repo" user-level command. There is an admin tool for serverside use, osc-admin which can force a repo publish but it's not something we can use.

This is highly undesirable, but I think if we downloaded the two "-0" source files, renamed them to "-1" and edited the DSC file to say "-1" to force a complete upgrade of the package it might work? The system would see this as "new sources" and trigger it's normal, default new release behaviour and... perhaps work. There's just sort of a complete lack of any helpful information coming up about solving this specific problem or why it's even happened.

terackspace commented 1 year ago

A coworker has found this is something going wrong with mirrors - finding another one directly, we can test it and get the newly updates SHA256 like so:

$ curl -O https://ftp.lysator.liu.se/pub/opensuse/repositories/home:/holland-backup/Debian_10/Packages
$ curl -O https://ftp.lysator.liu.se/pub/opensuse/repositories/home:/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb

$ grep SHA256 Packages 
SHA256: 1f825c5d08d857f4ee9cc1a51f4cce61625ddd673b1230d42cd4bf020fd4b118

$ sha256sum holland_1.2.10-0_all.deb 
1f825c5d08d857f4ee9cc1a51f4cce61625ddd673b1230d42cd4bf020fd4b118  holland_1.2.10-0_all.deb

We then see that the "invisible" mirrorcache service could be pointing us at a troublesome mirror without it being obvious:

# curl -IL https://download.opensuse.org/repositories/home:/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb

HTTP/2 302 
location: https://mirrorcache-us.opensuse.org/repositories/home:/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb

HTTP/2 302 
location: https://mirror.rackspace.com/openSUSE/repositories/home%3A/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb

We'll reach out to the mirror.rackspace.com admin and ask if they can do a forced-resync for us, as we see the other mirror properly updated and working.

tmrsiteops commented 1 year ago

Thank you, I appreciate you looking into this!

terackspace commented 1 year ago

@tmrsiteops give it a shot how, the mirror team did a forced resync and I see the two test items now match:

$ curl -O https://mirror.rackspace.com/openSUSE/repositories/home%3A/holland-backup/Debian_10/all/holland_1.2.10-0_all.deb
$ curl -O https://mirror.rackspace.com/openSUSE/repositories/home:/holland-backup/Debian_10/Packages

$ sha256sum holland_1.2.10-0_all.deb 
0b93750c21fbed005105689dcb1767427e1b15787fce1d0566de63fe51456645  holland_1.2.10-0_all.deb
$ grep SHA256 Packages 
SHA256: 0b93750c21fbed005105689dcb1767427e1b15787fce1d0566de63fe51456645
tmrsiteops commented 1 year ago

Unfortunately I'm still getting the same error output from apt-get install holland as before. But when I run your commands above I do see the matching hashes. I'll try again tomorrow and update here.

terackspace commented 1 year ago

Unfortunately I'm still getting the same error output from apt-get install holland as before. But when I run your commands above I do see the matching hashes. I'll try again tomorrow and update here.

I suspect (learning as we go) that if one mirror had an issue with not picking up a file that was updated but the name didn't change, that others might as well. When you get around to doing your debugging, try and run it in some sort of verbose/debug mode to observe which mirror it's getting content from. As just a test only, you could try manually updating your sources.list entry for Holland to point directly at once single mirror as well to go around their mirrorcache magic redirection service to help narrow in on the problem.

tmrsiteops commented 1 year ago

I just tried the install process using https://mirror.rackspace.com/openSUSE/repositories/home%3A/holland-backup/Debian_10/ manually added to my sources.list and it completed with no issues. Strangely, I first tried the install again following the regular install instructions and the apt-key add step returned gpg: no valid OpenPGP data found, which was not happening yesterday.

Thanks again for your help with this.

terackspace commented 1 year ago

The key seems OK when I manually check it, like so:

$ curl -O https://mirror.rackspace.com/openSUSE/repositories/home%3A/holland-backup/Debian_10/Release.key
$ gpg Release.key 
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa2048 2018-10-31 [SC] [expires: 2025-04-30]
      9CD2A8B2D32F27374937002E71BE948B04B476D3
uid           home:holland-backup OBS Project <home:holland-backup@build.opensuse.org>

That's the newer one we were trying to re-sign all the repos with before this issue turned into an ordeal. Given that the Rackspace mirror admins did a forced resync of the openSUSE content, it makes me want to believe that the upstream data is correct - but we might have downstream mirror problems not picking up files which were updated but have the same name.

Only other option left I can think of is what I outlined above as an undesirable - but valid - choice; bump the tarball + DSC to a new release tag to force all new filenames to appear in the repos. Doesn't feel right to me though, but if that's our only choice left... shrug

jrdnyquist commented 1 year ago

So I am a bit lost at sea here. Can you tell me how to:

A) How the install process differs from here:

https://hollandbackup.org/INSTALL.html (mostly concerned with Ubuntu and CentOS)

B) Can I just update the key on current installs?

terackspace commented 1 year ago

Disclaimer: I am not a Holland project member, but have access to the OBS project so I jumped in to help try and solve this problem. We are specifically focusing on the APT (Debian/Ubuntu/etc.) repos here in this Issue.

A) How the install process differs from here: https://hollandbackup.org/INSTALL.html (mostly concerned with Ubuntu and CentOS)

It should not differ - the intent/spirit of this work was to simply update a GPG key that signs the APT repos.

APT design signs a repository but not the individual packages; RPM design signs the packages, not the repository. When an APT repo GPG key expires/breaks/etc., it affects the entire apt subsystem (errors out) whether or not any software is installed - just having the repo present with a bad GPG key causes problems.

This is what we're fixing here - trying to re-sign the APT repos with an updated GPG key. There appears to be no direct way to do that in OBS, so we had to rebuild all the DEB packages, which as a side effect updated the GPG key signing all the repos. There was no new Holland release of software, leading to...

B) Can I just update the key on current installs?

In theory and spirit for the APT repos, yes. The RPM repos hosted on OBS are not in scope of this Issue on any work I've tried to do, all of that has remain untouched. The EPEL Holland project builds/signs their RPMs so they would not be affected by this GPG key in the OBS project. If the OBS RPMs need updated, that needs to be a new Github Issue for someone to tackle.

jrdnyquist commented 1 year ago

OK cool, this worked on existing Ubuntu 22.04.2 LTS system:

wget -qO - https://download.opensuse.org/repositories/home:/holland-backup/x${NAME}_${VERSION_ID}/Release.key |apt-key add -

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). OK Thanks!

terackspace commented 1 year ago

Closing this up, hopefully we're all done this time.