Signed releases

[htuch]

Running https://github.com/ossf/scorecard against https://github.com/envoyproxy/envoy gives generally a high score, but we are missing signed releases/tags. The idea is to attach a GPG ASCII signature to the release/tag artifacts.

Arguably the benefit is marginal if we trust GitHub security (how many Envoy consumers will check this signature?), but this seems a reasonable defense-in-depth best practice to follow.

[rialg]

Hi @htuch, I have some questions regarding the key used to signed the releases. What type would you like it to be? For how long should it stay valid? What its UID should be?

Following the Debian's guide on how to create signed github releases, I manage to put together a demo/POC on my forked repo.

BTW, could I help you with this issue ?😁

[htuch]

Thanks @grial1. TBH I'm not sure how we want to deal with key management. @alyssawilk @mattklein123 since you folks tend to cut the releases, is there a preferred methodology around key management and signing?

[alyssawilk]

No preference from me. If @grial1 is up for helping out, maybe he can doc up his PoC steps in the GOVERNANCE release instructions once we get the questions settled.

[rialg]

Sure 🙂 . Once decided how to handle the key management, I can submit a PR to GOVERNANCE.md with the steps to follow to create a signed release.

[mattklein123]

Once decided how to handle the key management

Can you summarize what kind of key management is required? Is this a flow that someone runs on their dev machine? Or something else? What type of key(s) need to be stored? I think a bit more details would help. Thank you.

[rialg]

For instance, when creating the key, as explained by this guide, an UID is needed to be provided. Which should be that UID? Then, the key needs to be uploaded to a public keyserver, so, which one should I use? For the example I uploaded the key to pool.sks-keyservers.net.

[htuch]

I think we might need some maintainer oriented flow around a shared signing key, maybe backed with Lastpass. But there are probably alternatives, e.g. a signing VM.

[mattklein123]

I think we might need some maintainer oriented flow around a shared signing key, maybe backed with Lastpass. But there are probably alternatives, e.g. a signing VM.

A signing process that is built into a VM or cloud hosted workflow (lambda, whatever) would be optimal, but a lot more work unless there is some service out there we can use for this (cc @caniszczyk who might know of something). In the interim, we could host the key in our shared LastPass per @htuch. I don't have any strong preference on key server for the public key. This is not something that I know a lot about so I'm happy with whatever is standard here.

[dlorenc]

Many cloud providers support asymmetric signing. Here's GCP's version: https://cloud.google.com/kms/docs/asymmetric-encryption

If you're already using a cloud provider's IAM for maintainers somewhere, this is probably a straightforward method.

[asraa]

Could we add GCP's KMS support to envoy-ci and sign images via cosign in our postsubmit docker CI workflow? We'd already have the credentials in our CI workflow. This would attest that the images on dockerhub were produced via CI. Users can verify their images if they have envoy-ci's public key view access (or we can publish the pk on GitHub) (via cosign verify -kms gcpkms://projects/envoy-ci/locations/<LOCATION>/keyRings/<KEY_RING>/cryptoKeys/<KEY_NAME> envoyproxy/envoy-dev:latest )

Later on we could get maintainers to locally sign release images to attest the code built by CI is what we actually wanted to release (via private keys in GCP/LastPass or whatever).

[markmsmith]

I'd love to see signed releases (and ideally contributor commits) for this project.
If it's any help, here's an example of how nodejs lists out the release maintainers' keys (in addition to having them on the standard key servers):
https://github.com/nodejs/node#release-keys
and here's an example of how it's verified in the official docker alpine build for it:
https://github.com/nodejs/docker-node/blob/13106300db1b4ce862010e3b35b86cc25cbe4cf5/14/alpine3.13/Dockerfile#L37-L58

As another example, Hashicorp hosts this page in their docs with their gpg keys and verification process:
https://www.hashicorp.com/security#template-page-security:~:text=PGP%20Public%20Keys,Release%20Archive%20Checksum%20Verification

[mattmoor]

Not sure what Envoy is using for CI, but if it supports OIDC (like Github actions) then you can leverage Sigstore's keyless signing and forgo the key management altogether. Projects like distroless, which started with key-based signing are now signing things both ways.

[phlax]

@mattmoor - interesting - thanks for pointers - i will look into this

this issue is one that is, i hope, going to be addressed fairly soon

[caniszczyk]

Matt, if CG wanted to help put this together and show that it can work, we'd roll this out in CNCF as a recommended option

Do you have concrete examples to point too?

On Thu, Jul 28, 2022 at 12:51 PM phlax @.***> wrote:

@mattmoor https://github.com/mattmoor - interesting - thanks for pointers - i will look into this

this issue is one that is, i hope, going to be addressed fairly soon

— Reply to this email directly, view it on GitHub https://github.com/envoyproxy/envoy/issues/14076#issuecomment-1198396430, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSIL4A2MLAWCT2LQHBO3VWK3AXANCNFSM4T2CYHXA . You are receiving this because you were mentioned.Message ID: @.***>

-- Cheers,

Chris Aniszczyk https://aniszczyk.org

[puerco]

Kubernetes images are signed with Sigstore's keyless signing, we do not do any key management for them. We are already starting work to sign our binaries too. Here's the current plan for the files and binaries: https://github.com/kubernetes/release/issues/2618

I am happy to help over here too!

[phlax]

any help would definitely be appreciated

we also have developed some gpg-based python code for achieving self-signing in the release pipeline - mostly in anticipation of releasing binaries rather than containers - but also as discussed in the linked k8s ticket to ensure artefacts are stamped/validated throughout pipelines

as this is quite high priority for us we will probably use the gpg approach initially to get things moving, but would welcome a mechanism by which we can authorize binaries as CNCF-signed - so open to other approaches, and i would be happy to follow up on this once we have release pipeline in place

[mattmoor]

✨ If you haven't seen the keyless approach, it is pretty magical ✨

@puerco did a lot of the signing work for K8s, so I cannot think of better hands to be in. 🤩

[wrowe]

Please note, signing a "package", e.g. a source or binary tarball/zip file/docker image, doesn't intersect at all with Windows "Code Signing", these are two different and distinct topics. Just noting here that these need to be resolved independently. https://github.com/envoyproxy/envoy/issues/15411