Support for reading pipeline secrets from Secrets Manager

[holmesjr]

Rather than writing secrets into a bucket with the default KMS key, it would be great to have support for Secrets Manager.

[holmesjr]

Perhaps it can be handled here: https://github.com/buildkite/elastic-ci-stack-s3-secrets-hooks/issues/21

[holmesjr]

Ok, I have a basic version of this working. It's stripped down to the minimum so we can work on some of the interesting issues.

One issue is how to specify which secret(s) to pull down. In the S3 plugin, we can rely on a naming convention. Secrets get a randomised string at the end of their ARN. Any ideas are welcome.

[lox]

Awesome! Is there a WIP somewhere that I can look at?

[holmesjr]

I'll make gists of the files I created. I was able to do it with a custom bootstrap, an env hook and a script that acts as the credential provider. I'll post links here.

[holmesjr]

The files are:

• bootstrap
• env
• creds helper

[lox]

Did you do any thinking about what format / structure of key you'd expect in secrets manager?

[holmesjr]

Ah yeah, that would have been useful info. I currently have this:

{
  "Cred": "https://USER:TOKEN@github.MYORG.com"
}

[holmesjr]

So it's just the git creds line as per the S3 plugin, under a "Cred" key.

[lox]

@holmesjr I spent a bit of time today expanding on those gists and wrangled it into https://github.com/buildkite/elastic-ci-stack-secrets-manager-hooks.

The env support isn't implemented yet, and I rolled back the "use a docker image" bit. Interested in your feels @holmesjr.

[holmesjr]

Does that mean they released a version of the AWS CLI that supports Secrets Manager into the Amazon Linux yum repo?

[lox]

I haven't checked yet, but I'll make sure the new elastic stack updates it's aws cli. The idea of depending on a third-party docker image makes me really twitchy.

[holmesjr]

You'll probably need to install pip and update the CLI that way.

[zsims]

Secrets get a randomised string at the end of their ARN. Any ideas are welcome.

You can still find them by secret key id (without the random parts). The random suffix of the ARN is only important for I AM resources.

FWIW we use AWS SM via a Buildkite plugin: https://github.com/seek-oss/aws-sm-buildkite-plugin (which is a different use-case to this issue)

[lox]

Secrets get a randomised string at the end of their ARN. Any ideas are welcome.

I didn't run into that issue, I wonder what I did differently 🤔

[holmesjr]

Did you perhaps use the id rather than the ARN?

[lox]

Ah, yes!

[holmesjr]

That will work for anyone with secrets in the same account, but not cross-account. Maybe it's not a big deal and we can use a secret naming convention.

[lox]

Yeah, secrets in other accounts is something that is harder to do with this model. I think we'd probably have to do something like store a map of secret keys in a known key on the local account, e.g elastic-ci-stack/mapping with something like:

{ 
  "elastic-ci-stack/global": "arn:aws:secretsmanager:ap-southeast-2:xxxxx:secret:buildkite-github-creds-xxxxx" 
}

That way you could map certain keys onto other accounts and keep the inference model, which I kind of like.