[Fargate] [Secrets]: Dynamically pushing secrets to Containers

[rohanmangal]

Tell us about your request Would like to get secrets dynamically updated in containers, when they are rotated in respective Secret Manager or SSM Parameter Store.

Which service(s) is this request for? This could be Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently when secrets are rotated in secret Manager or SSM Parameter store, the container will not receive the updated value automatically. Either task refresh or force service update is required as secrets are pulled when task starts. Would like to get secrets dynamically updated in containers, when they are rotated in respective Secret Manager or SSM Parameter Store.

Are you currently working around this issue?

Additional context When running tasks at production scale of 1k or more tasks, its becomes unreasonable to refresh or update the task for fetching new credentials.

Attachments

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html

Sensitive data is injected into your container when the container is initially started. If the secret or Parameter Store parameter is subsequently updated or rotated, the container will not receive the updated value automatically. You must either launch a new task or if your task is part of a service you can update the service and use the Force new deployment option to force the service to launch a fresh task.

[FernandoMiguel]

I understand your need, but this an issue that is terribly hard to change. It goes deep down to the way processes are executed in a docker container. Secrets are injected as environment variables. And usually the process executed in a docker container is PID 1. That means that it reads vars on start up and since it is PID 1, it can't be restarted. Restarting that process literally kills it, and the orchestrator will discard that docker as unhealthy and replace it, potentially causing downtime if there aren't other tasks to receive traffic

[FernandoMiguel]

The other approach that the community has taken is using side cars, like envoy. In that case the app talks to the side car to request credentials, and the site car takes care of reaching the creds server (like hashicorp vault or parameter store) and refresh the secrets.

[whereisaaron]

@rohanmangal Kubernetes supports dynamic binding of file-based Secrets within containers, so if you deploy to EKS you can use that to rotate secrets without restarting containers. You can further use the kubernetes-external-secrets operator to dynamically bind EKS Kubernetes Secrets to AWS Secrets Manager.

For both ECS and EKS you can use small sidecar containers as @FernandoMiguel suggests, though that is admittedly quite an overhead on 1k replicas.

[rohanmangal]

Thanks @FernandoMiguel and @whereisaaron

I understand your need, but this an issue that is terribly hard to change. It goes deep down to the way processes are executed in a docker container. Secrets are injected as environment variables. And usually the process executed in a docker container is PID 1. That means that it reads vars on start up and since it is PID 1, it can't be restarted. Restarting that process literally kills it, and the orchestrator will discard that docker as unhealthy and replace it, potentially causing downtime if there aren't other tasks to receive traffic

Definitely restarting the container is not a good idea, just to refresh credentials

The other approach that the community has taken is using side cars, like envoy. In that case the app talks to the side car to request credentials, and the site car takes care of reaching the creds server (like hashicorp vault or parameter store) and refresh the secrets.

Sidecar approach is good but it will add extra moving piece in service and would be too much(in terms of resources and management overhead) for retrieving new credentials values.

---

I was thinking of a more native way of refreshing credentials, for example: What if Dynamic Secrets can be injected to containers like task role is passed today by using AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

Then applications can curl URI and get the latest secret, without calling GetSecretValue API directly or using sidecar containers.

Thoughts?

[FernandoMiguel]

@rohanmangal that would mean your application would have to be smart enough to know how to use external secrets. That would add complexity to all your apps.

But can be done. Just add aws sdk and access parameter store directly from your application (given your example of many hundreds dockers you would probably hit the api limit rate)

[tinducvo]

I am very interested in this; would make rotation so much simpler.

[josiahwitheford]

This feature would be awesome for us as well. Currently we are looking into the sidecar idea as a way of updating the secrets as we don't really want to couple all of our applications with the secret manager implementation.