search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
264 stars 59 forks source link

Signing for release artifacts #187

Closed bgilbert closed 5 years ago

bgilbert commented 5 years ago

Set up signing for release artifacts.

Open questions:

Predecessor ticket: https://github.com/coreos/fedora-coreos-tracker/issues/87 COSA discussion: https://github.com/coreos/coreos-assembler/issues/268

dustymabe commented 5 years ago

I can try to pick this one up if no one else is.

dustymabe commented 5 years ago
  • What format should signatures take?

@bgilbert - Let me try to understand what we want: Here are a few options:

  1. sign the artifact itself and deliver detached signature (probably slow because need to shove the entire artifact over to the signing server):
    • fcos-XYZ.qemu.qcow
    • fcos-XYZ.qemu.qcow.sig
  2. sign a checksum of the artifact and deliver checksum + detached signature:
    • fcos-XYZ.qemu.qcow
    • fcos-XYZ.qemu.qcow.CHECKSUM
    • fcos-XYZ.qemu.qcow.CHECKSUM.sig
  3. deliver checksum file with inline signature of checksum file text:
    • fcos-XYZ.qemu.qcow
    • fcos-XYZ.qemu.qcow.CHECKSUM -> contains inlined ASCII signature

My understanding is that we want this for each artifact rather than for a collection of artifacts. i.e.:

  1. deliver checksum file with inline signature that contains checksums of all artifacts for build:
    • fcos-XYZ.qemu.qcow
    • fcos-XYZ.iso
    • fcos-XYZ.openstack.qcow
    • fcos-XYZ.CHECKSUM -> contains checksums of all artifacts + inlined ASCII signature.

4. is closest to what the rest of Fedora is doing today

ajeddeloh commented 5 years ago

+1 for a sig for each artifact. Option 4 is super annoying from a user perspective. If we can get option 1 to work with the fedora signing infra that'd be best in my opinion.

dustymabe commented 5 years ago

If we can get option 1 to work with the fedora signing infra that'd be best in my opinion.

but isn't that a bit of a waste on both the server and the client ? i.e. computing the data for a hundreds of MiB on both sides vs a small amount of data?

ajeddeloh commented 5 years ago

Computers are fast and gpg is hard. GPG verification isn't terribly slow or anything. I'd prefer a simpler approach (for users) over something faster but easier to mess up. I'm assuming the .sig would be a detached signature so it would be a trivial amount to download.

bgilbert commented 5 years ago

I have a pretty strong preference for option 1. All of the other options are more inconvenient to verify. In addition, the stream and release metadata should include content hashes (https://github.com/coreos/fedora-coreos-tracker/issues/98#issuecomment-496774211) and the bucket won't allow directory listing, so in order for users to find the artifacts they'll already have the hashes anyway.

dustymabe commented 5 years ago

cc @jlebon @cgwalters @lucab @arithx

cgwalters commented 5 years ago

All of the other options are more inconvenient to verify.

How's that? If we provide a checksum file it's literally just sha256sum -c --ignore-missing fcos-xyz.checksum plus gpg --verify. (Or replace sha256sum with sha512sum probably)

cgwalters commented 5 years ago

The Fedora signing server is definitely slow with large artifacts, it's a constant problem for RPMs, and I doubt it's going to be improved anytime soon as it's ultimately backed by hardware, not amenable to scaling, etc. Why force some slow hardware token to compute RSA over 600MB when you could just give it < 1k?

cgwalters commented 5 years ago

The signed checksum file approach is nearly a "rough consensus standard":

https://www.debian.org/CD/verify https://www.qubes-os.org/security/verifying-signatures/#3-verify-your-qubes-iso https://getfedora.org/en/security/ https://en.opensuse.org/SDB:Download_help

cgwalters commented 5 years ago

(The checksum file approach would require us of course to generate one from our meta.json but that's pretty easy)

ajeddeloh commented 5 years ago

How's that? If we provide a checksum file it's literally just sha256sum -c --ignore-missing fcos-xyz.checksum plus gpg --verify. (Or replace sha256sum with sha512sum probably)

That's... not great. People may not know about -c --ignore-missing and either verify by looking at hashes manually (garbage) or skip verification. I don't have those args memorized. I'd need to spend time looking through the man pages. If the barrier to entry is high enough, people will just skip verification, which is not good.

Why force some slow hardware token to compute RSA over 600MB when you could just give it < 1k?

Because I'd rather have us hit that pain than our users, especially if it just means taking a little longer on releases.

The signed checksum file approach is nearly a "rough consensus standard"

I'd argue it's not very good. Plain old gpg sigs are a rough standard on other (smaller) things we ship (etcd, ignition-valdiate, etc) and it'd be nice to have a consistent way to verify things.

bgilbert commented 5 years ago

100% agree with @ajeddeloh.

Why force some slow hardware token to compute RSA over 600MB when you could just give it < 1k?

It's computing RSA over a hash, in any event, and the hash should be computed off-token. I won't ask too many questions about that particular implementation, though. :confused:

Ideally we'd be able to compute the hash on our end and submit that, though I don't expect it. (Some PGP implementations can accept a hash directly; gpg won't.)

cgwalters commented 5 years ago

I don't have those args memorized. I'd need to spend time looking through the man pages.

But those 4 distribution links already document how to do this and work in very similar ways. (Interestingly, QubesOS does both approaches)

And let's continue:

OK, apparently Arch Linux does direct signatures.

The OpenBSD install guide explicitly mentions using -c --ignore-missing BTW too, although they invented their own not-GPG thing.

cgwalters commented 5 years ago

Ideally we'd be able to compute the hash on our end and submit that, though I don't expect it. (Some PGP implementations can accept a hash directly; gpg won't.)

Hum, but if the Fedora signing server is using a hardware token, maybe it supports providing the hash - if this is true actually it would also help for signing RPMs.

But I can say definitively that asking the current server to sign several gigabytes (particularly if done at any kind of frequency) is asking for a lot of pain. And the clear "rough consensus" here is to use signed checksum files.

bgilbert commented 5 years ago

But those 4 distribution links already document how to do this and work in very similar ways.

A bad interface is a bad interface, no matter how well documented or widely used. If we're telling people to type --ignore-missing we're doing something wrong. Every extra step adds friction that causes people to not bother verifying their download.

That also doesn't get into other use cases, such as 1) streaming verification, as needed by coreos-installer or user-built equivalents, and 2) verification from Python/Go/Rust without shelling out.

Aside from "we'd have to wait longer for signing", which is a tooling problem we shouldn't inflict on users, what's the actual downside to simplifying?

cgwalters commented 5 years ago

what's the actual downside to simplifying?

Inability to distinguish between "item corrupted in transit/on disk" versus "item has a bad signature".

bgilbert commented 5 years ago

There will be hashes in the release and stream metadata.

cgwalters commented 5 years ago

There will be hashes in the release and stream metadata.

That's true. On the other hand those are both our own bespoke/unique-to-us formats without a pre-baked command shipped in every operating system (and container base image, etc.) to use to verify. (On the other other hand, using JSON for this is sane and easy to parse, and more importantly extensible, so the decision to write those makes sense I think)

bgilbert commented 5 years ago

We will intentionally not have directory/bucket listings enabled for our release bucket, nor predictable URLs. If a user has obtained an artifact, they've gotten it one of two ways:

In the latter case, the download page can show the hash, or even synthesize a CHECKSUMS file for download.

cgwalters commented 5 years ago

You've put some thought into this and I see your perspective. Having a "smart" website would certainly be nice; not having a "dumb directory listing" is...an interesting call. There's some unique infrastructure here which on one hand is valuable, but on the other hand also unique.

I think I've established that most extant distributions are doing the checksums files, and a reasonable percentage of people who have used any of those other operating systems will find it completely familiar.

But eh, I've no opposition to it.

I think the pivotal thing really becomes the pain of using the Fedora GPG signer - and it's reasonable to consider that a bug to be fixed, but like I said I doubt it will be. I suspect other hardware signers are going to hit similar limitations (isn't fero also flaky?). Kind of the point of hardware signing is that it doesn't trivially scale :wink:

bgilbert commented 5 years ago

For context: the primary reason to omit the directory listing is to avoid having users or tooling implicitly grab the latest release (which circumvents our ability to control or stop rollouts) and to discourage them from grabbing stale releases. (A secondary reason is to avoid the Container Linux problem that every new artifact instantly became a compatibility constraint.) We want artifact access to always go through the stream metadata or something derived from the stream metadata. You're right that we're incurring tradeoffs in the process.

I think we should at least have a conversation with Fedora infra to see what's possible. If direct artifact signing is just completely infeasible, clearly we'll have to do something else.

Re fero: it has an unfixed bug that prevents it from being used to sign Go binaries, but it's been completely reliable for Container Linux signing.

ajeddeloh commented 5 years ago

To clarify: fero's problem is it can't sign large things. The CL signing process just signs a hash (not actually sure the details there) and small json files so it hasn't been an issue there.

cgwalters commented 5 years ago

Sorry I had only partially followed the design for the metadata; I apologize for needing to reiterate here the rationale. I understand the goals.

However, we're also simultaneous to this trying to get the RHCOS content showing up at https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/4.1.0-rc.4/ signed...and that is definitely going to use the same "signed checksum file" approach... And we're going to be training admins/tools to do that. Maybe in some future if this works out for FCOS we could try to adapt it for RHCOS too? I dunno...

To clarify: fero's problem is it can't sign large things.

Which is the same issue that the Fedora signer has...this should be some sort of...sign? :wink:

Anyways I withdraw my argument around CHECKSUM files, happy to proceed with the stream stuff. But that still leaves the issue of signing directly.

I also wanted though to follow up to:

That also doesn't get into other use cases, such as 1) streaming verification, as needed by coreos-installer or user-built equivalents, and 2) verification from Python/Go/Rust without shelling out.

RE 1, tooling can easily verify the checksum in a streaming fashion too right? Just pipe it to a checksum at the same time as downloading, this is quite standard. It's what libostree does downloading objects, etc.

As far as 2...I would strongly suspect that a lot of people would shell out to gpg2 --verify. There is an implementation of OpenPGP for Go I think (and rust) but it's a huge dependency to carry.

cgwalters commented 5 years ago

(The gpg dependency of course is why OpenBSD decided to go their own way, see also https://github.com/ostreedev/ostree/issues/1233 )

bgilbert commented 5 years ago

RE 1, tooling can easily verify the checksum in a streaming fashion too right?

Sure, it's still possible. It just makes the code more complicated: download the checksum file, verify its signature, grep out the line you need, replace the second column with -...

As far as 2...I would strongly suspect that a lot of people would shell out to gpg2 --verify. There is an implementation of OpenPGP for Go I think (and rust) but it's a huge dependency to carry.

Probably. I'd still prefer to avoid adding roadblocks for tools that don't.

dustymabe commented 5 years ago

I think we should at least have a conversation with Fedora infra to see what's possible. If direct artifact signing is just completely infeasible, clearly we'll have to do something else.

Is the above ^^ where we are at this point? Seems like we've got some voting for 1. and some voting for 4. Should I find out what the limitations are for 1. from Fedora infra and then we'll continue the discussion?

bgilbert commented 5 years ago

@dustymabe Yup, I think that's the next step.

dustymabe commented 5 years ago

I have added this as a topic for the fedora infra meeting tomorrow.

The meeting is on 2019-06-06 from 15:00:00 to 16:00:00 UTC in #fedora-meeting-1: https://apps.fedoraproject.org/calendar/infrastructure/#m22

jlebon commented 5 years ago

Is there a summary of the discussions from the FCOS community meeting and releng meeting? Or better, a link to the design doc :)

dustymabe commented 5 years ago

Is there a summary of the discussions from the FCOS community meeting and releng meeting? Or better, a link to the design doc :)

not yet. i'm working on something for it

dustymabe commented 5 years ago

In the meeting yesterday the infra team claimed that option 1. was feasible with the fedora signing server setup. I have created a Project Proposal for the infrastructure team so that we can get this work designed and scheduled

I have also created a few follow on tickets because discussing the design has identified more things that need to be done:

dustymabe commented 5 years ago

marking as blocked while we wait on feedback on the Project Proposal

dustymabe commented 5 years ago

I've added the Project Proposal to the meeting agenda for the fedora infra meeting today.

dustymabe commented 5 years ago

Discussed in the infra meeting. The proposed project is unlikely to get completed in time for the first preview release. A contingency plan right now is to collaborate with the infra team and get artifacts signed manually.

See https://pagure.io/fedora-infrastructure/issue/7884#comment-579900

dustymabe commented 5 years ago

This can be closed when the following are closed:

jlebon commented 5 years ago

@dustymabe I think #199 is separate from this ticket, right?

jlebon commented 5 years ago

Should we use the existing Fedora release signing keys, or our own keys? The former gives us key rotation for free, but we'd probably want to ensure we sign each release with the key corresponding to its Fedora major. That'd mean we'd need to synchronously switch keys as a rebase is promoted through the streams.

Just to record a decision here about this that was made OOB as well as rediscussed in the latest community meeting: we want FCOS to use the Fedora release signing keys.

We could make bumping needed configs part of the SOP for transitioning across major versions. Note also that the key for the N+1 release is shipped in the N release (for obvious reasons). Additionally, the OSTree remote we ship today accepts signing with any key in /etc/pki/rpm-gpg (something which I'm actually not a fan of, but works in our favour in this case).

Other things to note:

jlebon commented 5 years ago

Note also BTW that signing will be done in the pipeline itself via fedora-messaging, so we can trivially verify at build time that the right key was used.

dustymabe commented 5 years ago

@dustymabe I think #199 is separate from this ticket, right?

At this point I think the answer is yes. While we do still want the commits imported, that work is not required for this ticket to be considered complete.

When I wrote that I think we hadn't worked out all of the details and maybe I thought we needed to import the commit into the infra before we could sign it, thus it would be needed for this ticket.

dustymabe commented 5 years ago
  • We could teach RoboSignatory about the FCOS versioning scheme and make it use the right key from that.

That seems like a nice simple answer. Considering that we've done some work on integrating our signing into robosignatory and we're familiar with it a bit more I think this makes sense.

Note also BTW that signing will be done in the pipeline itself via fedora-messaging, so we can trivially verify at build time that the right key was used.

+1 for this too. On the robosig side we "detect fedora major release from version and use the corresponding key" and on the pipeline side we verify it was signed with the key we expected it to be signed by. That would give us confidence on both sides.

jlebon commented 5 years ago

I think we can close this now. Will open a follow-up ticket for checking at build time that the OSTree is signed with the correct key, and potentially making RoboSignatory smarter.

dustymabe commented 5 years ago

thanks @Jlebon