buildkite / agent-stack-k8s

Spin up an autoscaling stack of Buildkite Agents on Kubernetes
MIT License
78 stars 30 forks source link

Add GitCredentialSecret to checkout params #365

Closed DrJosh9000 closed 3 weeks ago

DrJosh9000 commented 4 weeks ago

What

Add a new checkout parameter, gitCredentialsSecret, that can be used to supply .git-credentials to the checkout container from a Kubernetes secret.

Screenshot 2024-08-19 at 3 50 06 PM

Why

Currently, checking out code with an SSH key loaded from a k8s secret is supported and even reasonably documented. The situation for HTTPS checkouts is much more lacking. I was able to get it working with a mish-mash of podSpecPatch and a pre-checkout hook, but I think that packaging it up into a single new parameter would be nice.

How

wolfeidau commented 4 weeks ago

@DrJosh9000 is there a way we can avoid writing this secret to the filesystem?

DrJosh9000 commented 4 weeks ago

@wolfeidau That's a great question. I did a little more digging and thinking.

AFAICT our options for accessing k8s secrets within a container are limited to:

  1. referencing a secret through a environment variables with env (or multiple at once, with envFrom)
  2. mounting a secret volume

Either way, the data is accessible within the checkout container in plain text. I went for option 2 since it works with git's built-in credential helper.

We could switch to option 1 (env vars), but would need some workaround.

Copying the file means it often lives beyond the checkout container, so a workaround for that would be to add another ephemeral volume for the read-write copy of the file (I plan to do this, if we stick with option 2).

Elastic CI Stack uses s3 secrets hooks, which among other things implements a custom git credential helper in Go. We could do something similar.

wolfeidau commented 4 weeks ago

@DrJosh9000 Maybe this is a thread to pull on separately, i think the best way forward would be a custom git credential helper of some sort, this way other stores like hashicorp vault, aws ssm or secret manager, could be used in the future.

Maybe something we attack in the agent proper so it is more widely usable and build on the implementation already there.