search

EmbarkStudios / k8s-buildkite-plugin

Run any buildkite build step as a Kubernetes Job
https://embark.dev
Apache License 2.0
49 stars 19 forks source link

Added External Secrets Support #68

Closed clbx closed 1 year ago

clbx commented 1 year ago

This PR adds arguments for creating ExternalSecrets object so that secrets from external secrets platforms (Vault, AWS, Azure) can be easily accessed from the CI

The majority of the heavy lifting is done by ExternalSecrets, these additions simply create a ExternalSecret resource, validate that the secret is created (fails the job if it was not) and then removes it once the job is finished.

I added an example to how this can be done in the buildkite yaml, but I'm open to suggestions on how to change it since I'm not particularly thrilled by the format the information has to be passed, but I could not find a better way to do so and I figured it best to follow the convention used by other arguments

clbx commented 1 year ago

So the main goal is to not have persistent secrets sticking around in the cluster and be easy to reference secrets in our Vault instance. We use ExternalSecrets in a lot of instances already so integrating it with this plugin made sense for our builds.

We have a lot of secrets in vault that we reference in different builds, these are logins for various services, cloud service accounts, etc. Its unrealistic to have them all sitting in kubernetes. Our developers do not have access to the cluster, but can create secrets in vault for whatever reason. This solves two issues our pre-k8s pipelines suffered from

  1. Developers can reference secrets without needing access to the cluster, and can easily create new secrets in Vault (or any other supported store)
  2. Secrets are short lived. Secrets only stick around for the duration of the job. Persisting secrets are carry a risk.

I definitely understand if you feel this adds too much secrets functionality and increases complexity. We have a huge use case for this internally and figured it would be beneficial to contribute it back.

tgolsson commented 1 year ago

Great, that's a good philosophical explanation too. I'll have a think about it - I'd never heard about this operator until now so I'm not sure about how ubiquitous it is. If it's commonly used and would benefit the wider community then it'd obviously be a good merge.

@keith Do you use/know about this operator (not necessarily for Buildkite, just in general)?

clbx commented 1 year ago

I've seen it used fairly often, as far as I know its a pretty common way to get secrets from an external source. Here is the their repository: https://github.com/external-secrets/external-secrets.

tgolsson commented 1 year ago

Thanks :-) I'm leaning towards a merge, just a bit swamped right now to think about the exact details and concerns. Will try to get to it next week.

clbx commented 1 year ago

No worries, we may have a few changes to the configuration coming down based on feedback from users.

clbx commented 1 year ago

My recent changes add mount-path-external-secrets so that a directory can be specified to mount the externalsecrets secrets to.

clbx commented 1 year ago

@tgolsson is there anything else to check with this or can it be merged?

tgolsson commented 1 year ago

I've just been super-busy, will try to get back to it soon.

clbx commented 1 year ago

We have some other features we want to introduce to our main branch, so I'm going to close this PR and open a new one with just the ExternalSecrets changes so future commits do not pollute this PR.