Open petemounce opened 6 years ago
Thanks for the writeup @petemounce! I'll copy my response to your initial email in here and then follow up on the bits that are new:
Securely running third-party builds is a really tricky problem that we've just started to think about in some depth. Generally our advice here is to have a block step at the start that checks a white list of repos and contributors and then do a code review before running the build, but eventually we'd like to have some good answers on how to have enough defense-in-depth in place that we feel comfortable recommending automated third-party builds.
As you've raised, if you run third-party builds on your buildkite-agent host, it's close to impossible to achieve a level of security. Thankfully we have containers on windows and linux. If you run your builds inside an ephemeral container, you have a much better chance of locking down what can and can't be accessed.
I just spent some time putting together https://github.com/buildkite/docker-bootstrap-example, which might be a starting point for you. It mentions two pieces of tech that are going to be a key part of secure, ephemeral third-party builds, namely an IAM metadata service proxy and sockguard. Check it out and we can discuss, with the IAM service in place (that can deliver limited AWS permissions to a single container) it would handle a lot of the secrets questions (via SSM Parameter Store).
One piece we don't have a good answer to (yet) is the Agent Access Token. At the minute the Agent registration token is used by the agent to connect to our API's, where it gets an Agent Access token, which is a session-based token for the agent's connection to our API's. Leaking the Registration key is a major deal, because it allows new agents to be registered, but the Access token is scoped much more strictly. Still, you could do enough things with an Agent Access Token, for instance restarting a build and injecting some new steps via a pipeline upload. I'd suggest not exposing the Access Token to third-party builds and perhaps deal with artifacts and metadata some other way.
Started work on an alternative to passing an Agent Access Token to jobs in https://github.com/buildkite/agent/pull/759.
Agent v3.1.2 has an experiment in it for passing a socket to builds vs $BUILDKITE_AGENT_ACCESS_TOKEN
, we're hoping this will become a standard feature in the agent soon!
This is mentioned in the other bug thread, but if you haven't seen this little tool, it's a great starting place for secure open source contributions. I highly recommend it for the time being. https://github.com/mvines/ci-gate
Oh, excellent. We'd been intending to write a plugin for the Kubernetes' team's workflow automator, prow (https://prow.k8s.io/plugins) if one didn't already exist. Basically, /ok-to-test
from a trusted contributor.
https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ is a good-sounding writeup of why to not use environment variables for secrets.
Yup, completely agree with that @petemounce!
Highly relevant: https://edoverflow.com/2019/ci-knew-there-would-be-bugs-here/
So, one use-case that I have is offering CI for the various open-source projects and products that improbable.io has and will offer. I assume I'm not unique.
Some requirements that I have:
contributors need to be covered by a contributor-license agreement.
contributions from untrusted contributors should be assessed for "safety" - since, obviously, continuous-integration is aka "Remote Code Execution as a Service". Examples of recon below. I'm sure we could collectively come up with others with some effort; penetration testing is entertaining. Thing is, I'm not really sure how to defend against things like this without also being extremely restrictive. It's a blacklist sort of approach, and that's inherently reactive. Probably there're the makings of a blog post as I do this...
I would very much like external contributors to be able to see the build logs, so that they can self-service debugging their PRs.
Some outcomes that I'd like to explore:
I'd be really interested to see some links to good reading on setting up a CI cluster that accepts external contributions but in as safe a way as possible, since this is exactly one use case that I'm going to be enabling.
I'd also be really interested to see to what degree people that have done that, have also set up the workflow for vetting external contributions, and then whether there are ways to track "trustedness" over time beyond subjective "well, they've done x PRs and look legit, let's tick the box now".
My context
I'm going to be running buildkite agents within GCE for Linux and Windows (on naked VMs first, then probably shifting to managed Kubernetes), and on-premise for macOS.
I'm likely to, later, spin up clusters of Linux and Windows within AWS (again, probably naked VMs first, but possibly skipping that if I got to Kubernetes within GCE earlier than I expect).
I'm very likely to be using Hashicorp Vault to give secrets to agents.
The on-premise estate will also include some Windows machines because we need to involve games consoles in CI, as well. This is a really entertaining prospect, too.
Recon/exploit examples
Brute force
Get me the registration token, now I can set up my own agents and start to steal IP...
(I'm pretty sure you've got this covered, since I've seen this value be
hunter2'd
)Default credential locations for cloud things
Both of these recons will probably get me at least read/write to the artifacts bucket(s), if successful, allowing it to be scraped or poisoned. Additionally, it would get me read/write to my Bazel remote cache, which would be Really Bad.
Probably also read-access to the secrets bucket if https://github.com/buildkite-plugins/s3-secrets-buildkite-plugin is in play, for the duration of the build running at any rate. I think the plugin could be improved by using session-tokens (so, the IAM profile allows the machine to assume a role that can read from the bucket, rather than allowing it directly) and then explicitly revoking the issued token inside the pre-exit hook? I recently discovered the AWS credentials files support role-assumption built-in (https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html), which would have saved me a tonne of typing if I'd realised that four years ago...
AWS
On a competently run AWS instance, this will get me that instance's IAM profile until the next 60 minute rollover (default, I think; might be out of date on that), but I might luck out and get an AWS instance that has been overprivileged...
GCE
On a competently run GCE instance, this will do similar:
Metadata services
Both clouds have unfettered access to metadata services that one can reach, from a shell. So, probably remove
curl/wget
. But... On a build machine? Probably need both of those, and that's a blacklist approach anyhow.Metadata services contain things like ssh-keys and cloud-credentials, as well as other metadata.
ssh keys
The buildkite agent docs guide how to set up ssh keys so that the agent can checkout from git etc, and mention
ssh-agent
only in the context of using more than one ssh key.I'm not an expert, but I think at the end of following that guide, a build script would be able to do:
and then start checking out my source code shortly thereafter.
I think I've read that it'd be possible for one user to give access to a running ssh-agent to another user, so that the 2nd user (here, buildkite-agent) doesn't have access to the keys at rest on disk. I think that would be a great addition to that documentation page - a directive to make the ssh private keys not readable to the buildkite-agent user, but have some other user mount them into the ssh-agent, and then share the socket to the buildkite-agent.
docker
Oh, docker. Apparently, docker installs by default to run with root privileges. I gather that might not be A Great Plan.
@lox pointed me at https://github.com/buildkite/sockguard which looks pretty promising.