search

concourse / registry-image-resource

a resource for images in a Docker registry
Apache License 2.0
89 stars 107 forks source link

feat: add support for dynamic credentials during get/put #322

Open lrstanley opened 2 years ago

lrstanley commented 2 years ago

This PR add support for using dynamically generated credentials inside of the get and put step. This is primarily beneficial when one is unable to generate long-lived credentials. For our specific usecase, any usage of AWS credentials must be short lived (less than 90min), and the way those credentials are generated does not integrate well with this resource.

I didn't add duplicate sections for those parameters under get and put definitions in the readme, as the ideal in most situations is for people to use the source configuration.

One item I'm not quite sure about is in relation to https://github.com/concourse/registry-image-resource/pull/96#issuecomment-603831222 (@vito) -- I am not sure if a resource is only used as a put (and the subsequent get), if this breaks the global shared resource model. Thoughts? Is there a better way around this issue?

taylorsilva commented 1 year ago

So this workflow only works when you're putting an image. It would never work if your job is only trying to get an image since the check step has no way of getting credentials passed in like a get or put step would. This feels a bit half-baked here because of that.

With rotating credentials you'd be resetting the resource configuration every time new credentials are passed into the resource, which would break the version history on the resource (if you care about that).

The only workaround I can think of at the moment is to have some other job fetch credentials and update the secret store you have concourse connected to with the new credentials every time the old one is about to expire. wdyt?

lrstanley commented 1 year ago

Yeah, I've been thinking about how to get those credentials into check steps in an elegant way for awhile, with no luck. Unfortunately, unless some of the future RFCs are pushed forward, I don't know if it'll be possible.

We had a few hundred users/pipelines using the approach of a job that runs every few hours, generates the credentials, write them to vault, so they can be pulled by concourse dynamically. The problem with that approach is that it's drastically more complex, users have trouble understanding the workflow, and it sends quite a few more requests to Vault (we already peak 40mil+ requests daily to our Vault clusters, which are used just for Concourse).

We are currently now using this PRs approach via my feature branch for all of those usecases, which seems to simplify getting this setup for most folks. It's definitely not perfect, though.