Need a way to use permission slip with buildomat

[labbott]

The current plan is to move to permission slip for storing all artifacts instead of a combination of github and a git repo. The eventual goal is to run permslip --url=$PERMSLIP_URL get-artifact <hash>. This requires access to get a token via OAUTH2 which can be saved to use with permslip login

(cc @plotnick @augustuswm @inickles)

[jclulow]

At least at present, buildomat intentionally lacks a classical "secret store" where people would insert hard-coded tokens. This pattern is not great for fairly obvious security reasons; e.g., a pull request from outside the organisation, or even some software supply chain attack, could trivially exfiltrate these secrets -- assuming they don't just end up in the job output itself under xtrace output from bash! Because it is a long-lived token that is entirely sufficient for authentication, it would be super easy for it to be stolen and kept and used by attackers much later on without us noticing.

For example, today, we use our GitHub Application credentials for the repository that owns a job to generate a time-limited read-only token of limited scope. That token is then available to the job for access to GitHub private repositories and to make otherwise severely rate-limited API requests, etc.

I have a couple of initial follow-up questions:

• Are the artefacts we're going to be fetching here actually private data? Could they be made accessible without authentication?
• If authentication is required, is it possible for us to store a long term refresh token in the buildomat system itself and use it to generate short-term read-only access tokens to inject into jobs, as we do with GitHub tokens?
• Would it be possible for buildomat to use an asymmetric key pair, such as a request signed with an ECDSA key, to obtain the short term credentials from permission slip? We could then keep the key in a HSM (e.g., a Yubikey or other PIV smartcard device), which would make it less likely that a compromise of the system itself would result in exfiltration of the refresh level credential we'd been granted.

[labbott]

permission-slip also works on GitHub access credentials. The trick is that the e-mail address associated with a token has to end in oxidecomputer.com . This seems like it could work but I'm also reaching the limits of my understanding of GitHub APIs.

[augustuswm]

In both cases we are effectively going to require a credential for a "user" account (of Google or GitHub) given that we use the verified email as the mechanism for authorization.

How difficult would it be to build out that second option? As that seems fairly ideal and flexible. Essentially in an abstracted way where buildomat can make a call to a configurable remote endpoint with proof of identity. Ideally something where buildomat signs its requests with a private key, and we can then have services that verify that signature and return tokens to be injected into the build process. It is then the responsibility of those services to ensure that the tokens are appropriately scoped and given short durations.

[inickles]

Are the artefacts we're going to be fetching here actually private data? Could they be made accessible without authentication?

My interpretation of the issue is we want buildomat to upload build artifacts to permslip (not just(?) download), which we'd definitely want auth for.

[labbott]

Are the artefacts we're going to be fetching here actually private data? Could they be made accessible without authentication?

My interpretation of the issue is we want buildomat to upload build artifacts to permslip (not just(?) download), which we'd definitely want auth for.

I was mostly focused on the downloading to replace what we currently have in https://github.com/oxidecomputer/omicron/blob/main/.github/buildomat/jobs/tuf-repo.sh to grab RoT/SP images but you raise a good point that I think we are going to want to have buildomat sign TUF repos in the long term too.

[inickles]

In both cases we are effectively going to require a credential for a "user" account (of Google or GitHub) given that we use the verified email as the mechanism for authorization.

:/ I'd prefer we don't, as this seems to me like a hack to work around how auth currently works in permslip. I'd rather we implement some kind of machine principle and/or service account in permslip, which I think we want for the device identity signing boxes in manufacturing anyway.

I'm not totally against this as a temporary solution, and I know we have some instances of it already with other systems, but I'd prefer we avoid it where we can.

[augustuswm]

I think that is where my understanding on permslip ends, if we can extend it to be able to working with a machine principle that would be even better.

[inickles]

Are the artefacts we're going to be fetching here actually private data? Could they be made accessible without authentication?

My interpretation of the issue is we want buildomat to upload build artifacts to permslip (not just(?) download), which we'd definitely want auth for.

I was mostly focused on the downloading to replace what we currently have in https://github.com/oxidecomputer/omicron/blob/main/.github/buildomat/jobs/tuf-repo.sh to grab RoT/SP images but you raise a good point that I think we are going to want to have buildomat sign TUF repos in the long term too.

Gotcha, thanks. Such signing keys would have to be long-lived, right? Unless we were able to separate between 'when it was signed' vs 'how long the signature can be validated for', which sounds familiar for TUF now that I say that.. Memory is fuzzy at this point.

[jclulow]

permission-slip also works on GitHub access credentials. The trick is that the e-mail address associated with a token has to end in oxidecomputer.com . This seems like it could work but I'm also reaching the limits of my understanding of GitHub APIs.

buildomat is a GitHub Application, and authenticates on behalf of the installation of that application.

In the context of a particular Check Run, we can update that check run as essentially failed, and include a (hopelessly constrained, in keeping with The Zen of GitHub) set of little "action" buttons that users can press. That's how we currently require an in-organisation use to authorise the running of jobs on a PR from outside the organisation. Every time the user presses an action button, we get a webhook from GitHub that includes the GitHub principal of the user who pressed it -- we use this to decide if someone is authorised or not. We can't, though, perform any authenticated actions on behalf of that user; we're merely told about them, we don't have any kind of OAuth context for them.

Gotcha, thanks. Such signing keys would have to be long-lived, right? Unless we were able to separate between 'when it was signed' vs 'how long the signature can be validated for', which sounds familiar for TUF now that I say that.. Memory is fuzzy at this point.

My assumption is that any signature buildomat generates would be performed inside the permission slip secure context. TUF repos are astronomically large, at least compared to Hubris artefacts, so my assumption is that we would be submitting some proxy for the contents; e.g., a content hash of the TUF repository file, as well as any policy-relevent chunks of metadata from inside the archive, and permission slip would sign something that includes that content hash. Consumers would verify the authenticity of the certificate and then use the contained content hash to verify the integrity of the TUF repository. But there are also many ways to skin that cat and I don't think it's especially relevant here, except to say that either:

• we send all the signed stuff to permission slip, and it manages the signing key (seems preferable)
• we provide a way for buildomat to request a short-lived intermediate CA for signing artefacts on its own, and some library for doing so appropriately (this feels probably not ideal)

How difficult would it be to build out that second option? As that seems fairly ideal and flexible. Essentially in an abstracted way where buildomat can make a call to a configurable remote endpoint with proof of identity. Ideally something where buildomat signs its requests with a private key, and we can then have services that verify that signature and return tokens to be injected into the build process. It is then the responsibility of those services to ensure that the tokens are appropriately scoped and given short durations.

I think this sounds about right to me. The buildomat system can have a private key (ideally in a HSM) that identifies it as buildomat, and permission slip can have the public key on record. We would make some request to permission slip requesting a short-lived access token with a particular scope, signed with our key, and then we would inject the access token that we get back into the job for use with the permslip tool. The scope could easily include, today:

• the GitHub Application installation ID, which should include the owner of the installation (in this case, the oxidecomputer organisation)
• the GitHub Repository ID, an integer which uniquely identifies the repository; we can also include the name, but that can obviously change due to renames, though that's probably unlikely in practice
• the GitHub user who owns the pull request or who made the push to the branch that the job was triggered by

Note that all of this information is stuff that we'd be getting from GitHub, basically, via web hooks (which are themselves signed by GitHub). You'd be trusting buildomat not to lie about or mishandle their contents. This seems reasonable for having permission slip sign development or internal use artefacts, where we don't currently require multiple approver flows.

For production/release artefacts I assume we'll do something else; e.g., upload an artefact to permission slip as a request for something to be signed and then the job will be finished. The approvals process and subsequent actual signing is presumably out of scope for the buildomat job itself?

[inickles]

For production/release artefacts I assume we'll do something else; e.g., upload an artefact to permission slip as a request for something to be signed and then the job will be finished. The approvals process and subsequent actual signing is presumably out of scope for the buildomat job itself?

Yeah, that makes sense - the buildomat job would be done, the "actual signing" here refers to Release code signing keys (RFD 286), and humans using permissionslip are responsible from there on.

My assumption is that any signature buildomat generates would be performed inside the permission slip secure context.

I could see that being the case, though, and just as a way to skin a cat, I could also see wanting to have a buildomat job create a signature for the artifact it build, so that others, not just permissionslip, could track the origins of an artifact to the specific environment it came from, using a similarly published pubkey as the one used for permslip auth. If permissionslip is the only system that will ever care about that, then sure, it can track that in its database from context an artifact was uploaded.

[plotnick]

We decided in the Firmware Release Engineering Sync that permslip should support anonymous authentication and public-key authentication for read-only and read/write use cases, respectively. The first will require no support from buildomat, the second a bit of support.

[plotnick]

Fixed by permslip#149. Buildomat can now use either anonymous or sshauth authentication to talk to a permslip server. Please open follow-up issues for authorization policy changes as needed, or comment here if there are any questions.