[EKS] [request]: AWS Secrets Manager / SSM Parameter Store

[jaronsummers]

Tell us about your request It would be super great if things from secrets manager and/or parameter store could be exposed as secrets within Kubernetes.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We're in the middle of a somewhat stilted migration into Kubernetes, and there's not a ton of confidence that Kubernetes is the forever-future, and being able to manage secrets outside of kubernetes while accessing them natively would be very pleasant in terms of providing additional flexibility and making for a softer landing for devs.

Are you currently working around this issue? We are using native k8s secrets encrypted + stored in SCM, but that is probably not the future. I'm going to write + test a lambda that will push secrets into kubernetes, but I am unsure of how to manage namespacing and permissions, and how to do all of this securely. A lambda will also be complicated because it will either be one-way, have to maintain state to know which secrets have been removed and should be removed from kubernetes, or be the so-called single source of truth, and delete ALL secrets that aren't in secrets manager/ssm/whatever, which from a hygiene perspective sounds great, but from a dev experience perspective, much less so.

[granite-cloud]

I am looking at the same scenario and would like to be able to use different backends (in helm charts or native k8s secrets) to pull in sensitive values stored outside k8s. This would be similar to confd which has a backend for param store and secrets manager amongst others with templates .

It is very common to have a single solution for secrets that would be nice to integrate with k8s. (Hashicorp vault or Aws services like param store/secrets manager)

[benderillo]

I second this request. Would be awesome to have this feature! We are storing a lot of information about infrastructure settings that needs to be accessed by services deployed on EKS. Having the parameters available via EKS secrets would make it so much easier.

[arvinep]

Is this feature somehow scheduled for any release? In which release we can expect it?

[whereisaaron]

@arvinep this is in the ‘proposed’ column on the roadmap, so it has not progressed beyond a community suggestion. So please thumbs-up the original post above to encourage AWS to consider it.

[granite-cloud]

Here Is an example of using a custom resource definition to integrate KMS for secrets management from godaddy. There are also some links to other projects that offer integration with param store and vault. This may be helpful to some as a better way to have a single source for secrets that is external to k8s. ( At least until this functionality makes its way into eks eventually. ) I have not had time to at least play around with any of these yet.

kms example from godaddy

[jfgudk]

I think it is critical for EKS to have a methodology that allows secrets to be consumed without storing them unencrypted. All of the solutions I have seen either store the secret in k8s secret which is not encrypted at rest or require the use of an init container.

I think the best solution would to extend the k8s secret to dynamically read from AWS Secrets Manager on demand. Also, hopefully this will also allow the use of the planned pod level roles to ensure appropriate secret access controls.

[dikylung]

this would be an ideal scenario

[tabern]

@jfgudk have you taken a look at this project? https://github.com/kubernetes-sigs/aws-encryption-provider

This provides KMS for K8s secrets but is not the external secret store integration requested here.

@jaronsummers adding this to our research list - solid proposal.

[jfgudk]

@tabern Thanks for the response. That project looks like I would need to modify the API servers and their startup settings and run pods on them which we are not capable of doing with EKS correct? At least part of that solution also requires k8s 1.13 and EKS is at 1.12. I am pretty sure AWS is actually encrypting the etcd database itself already.

My suggestion here and general issue is around integration with AWS Secrets Manager in order to comply with in totality all the following generally accepted universal secrets management requirements:

• Secrets are encrypted at rest and in transit
• Use of secrets is logged and auditable
• Secrets can be rotated and revoked
• Access to secrets can be controlled

Ideally we also would like to have a portable and backwards compatible solution with current k8s deployments and common practices.

[mycrEEpy]

There is an implementation by GoDaddy for AWS Secrets Manager: https://github.com/godaddy/kubernetes-external-secrets

[jweite]

I'll add a +1 on this. IHAC who needs to run pods in both AWS and non-AWS environments, and would benefit from being able manage the secrets for their AWS environment in AWS Secrets while still retaining the portability that the standard K8s Secrets mechanisms provide.

(They'd benefit from being able to manage their non-secret parameters in AWS Parameter Store too, integrated transparently to K8s ConfigMaps).

[pksinghus]

There is an implementation by GoDaddy for AWS Secrets Manager: https://github.com/godaddy/kubernetes-external-secrets

This creates a k8s secret which is just base64 encode of clear text. Which isn't a secret at all. Negative on this one.

[jfgudk]

As of k8s 1.13, encryption of secrets at rest is supported https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ - the question is whether EKS has enabled this.

Also note that etcd itself is also encrypted (prior to 1.13) - https://github.com/aws/containers-roadmap/issues/263

Access to k8s secrets can be granularly controlled and audited - with control plane logs sent to cloudwatch.

With these elements in place I don't see why it would not be viable to use k8s secrets to pass thru AWS secrets to pods to maintain cross platform portability and still maintain acceptable access/audit controls.

[pksinghus]

Encryption at rest alone managed by access control of secrets. How about pod fetching the secret using IAM role at startup/using initcontainer and mounting it on pod?

[jfgudk]

That is an option, but it means your applications are not portable.

The portable solution is for pods to consume k8s secrets as env vars or volume mounts.

[pksinghus]

But k8s secret isn't much of a secret. Base64 decode lets everybody look at the secret. Your password is more exposed now than before going to k8s. And you can't even move the secret/configmap into its own namespace because it's not possible.

[jfgudk]

And anyone who has the privs to deploy in your cluster has access to deploy a pod with the IAM role and access the secret anyway. Your cluster workers and any pod can call AWS API to get your AWS secret in clear text. They either have this directly or have the privs to assume an IAM role that does.

[granite-cloud]

Yes I agree on the sentiment around K8 secrets not being a great implementation. This is why I thought it was helpful when ECS allowed a task definition to directly fetch a secret from param store or secrets manager and use as env var. It would be nice if we could do something similar with K8, as was described above....

[jfgudk]

You can put secrets and config maps in any namespace and restrict access to non-system admins, at the pod level.

[gatesjd]

Please also take a look at this blog post for some interesting ideas. It describes a solution that injects secrets from vault into K8s pods--without them passing through etcd, and without a daemon.

To me the design has some distinct advantages, as well as some tradeoffs; but I didn't see a reason you couldn't use a similar mechanism to inject secrets from AWS services.

Whatever you end up with, please make sure it also handles dynamic secrets, like database credentials.

[inductor]

I need this >_<

[pksinghus]

I need this >_<

It is available. Now you can assign IAM roles to pods. Use SDK to query AWS as much as you want.

[inductor]

@pksinghus I’m pretty sure you’re just trolling but why on earth is this issue still not closed if it’s available

[pksinghus]

@inductor They have done all they can for you to use Secrets Manager or any other AWS services - https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ They will get around to closing it soon. Thumbs down doesn't change the facts :)

[jaronsummers]

Yes, everyone knows they can modify their code to retrieve the values directly from secrets manager, thank you for not helping in any way.

The point you are intentionally missing is that kubernetes actually already has an idiomatic way to provide secrets to the execution environment, and first party support for that would be nice.

[pksinghus]

Kubernetes also doesn't stop you from writing service operators and init containers.

[jfgudk]

Think portability.

[toricls]

@pksinghus I believe what people here want to see from AWS is something like a native-ish integration between K8s secrets and AWS Secrets Manager, to use K8s secrets just as an interface without installing/managing controllers or something by ourselves. So your proposal is absolutely right when we talk about apps in containers which just consume AWS services like S3 or DynamoDB as part of its business logic, but I think we need this issue keep opened from the perspective above.

[jfgudk]

I think the best solution would to extend the k8s secret to dynamically read from AWS Secrets Manager on demand. Also, this should allow the use of pod level roles to ensure appropriate secret access controls and auditing for traceability.

Without something like this, we might as well not use secrets manager at all and use native k8s secrets populated by some other mechanism.

The value secrets manager gives is

• Secrets are encrypted at rest and in transit
• Use of secrets is logged and auditable
• Secrets can be rotated and revoked
• Access to secrets can be granularly controlled

These things are needed for regulatory compliance purposes. If we're not getting that value why would we pay to use it?

[pimperator]

I am also very interested in having this solved! Just like having in ECS "ValueFrom" and an existing ssm arn

[miguelaustro]

Any progress on this?

[TarekAS]

I think the CRD approach would be the best solution. No need to extend the Secret resource.

This is how we're using godaddy's kubernetes-external-secrets to manage our secrets.

First we create the actual secret in SecretsManager via Console or CLI (which would allow us to use IAM to control access). Then we safely commit to our app/chart repo an ExternalSecret object that references the secret. The ExternalSecret controller will fetch the secret from SecretManager and will create a corresponding Secret object which you can use.

In my opinion this is a better GitOps workflow than using git-level encryption such as SOPS, since I consider secrets to be data, not code. In addition we can leverage the SecretManager's other features for things like rotations, IAM, RDS integration, etc.

It would be great if we get an official CRD that works like this.

[jaypipes]

Would https://github.com/deislabs/secrets-store-csi-driver be of interest here? Specifically, AWS adding support for an ASM provider driver?

[cscetbon]

@TarekAS how do you handle credentials rotation ? I suppose you have to recreate the ExternalSecret object every time you update it in AWS Secret Manager ?

[miguelaustro]

I implemented kubernetes-external-secrets, but it still relies on Kubernetes Secret, that means secrets are base64 encoded, not encrypted. I tried adding KMS, but the implementation was a bit hard. So far, let pods to assume a role via the aws "OpenID connect" Identity provider with a restrictive role to AWS Secret Manager items, seems to be the best end-to-end security solution, but not many programmers want to add AWS SDK. A native solution to safely inject variables from AWS SM to pods is still pending, from my PoW.

[Moulick]

Same way we are doing, using ASM to store secret, kubernetes-external-secrets to sync to cluster and commit the externalsecret to the gitrepo. Only problem is that this is a one way sync, from ASM to k8s. Would be nice to have a 2 way sync.

[DekusDenial]

How would the shipping of https://github.com/aws/containers-roadmap/issues/530 affect the status/progress of this request?

[inductor]

@Moulick One thing I am aware about that OSS is that the Kubernetes Client that they use is not very well maintaned.

[alimeerutech]

+1

[Moulick]

@inductor Might be. But what I can say is that it is working almost perfectly for about 2-3 months now and using it with https://github.com/stakater/Reloader works out awesome.

[rverma-jm]

+1 for secrets csi Driver. Already vault and azure released an implementation over it. https://github.com/Azure/secrets-store-csi-driver-provider-azure

The only disadvantage I see is that we need a daemonset, also I am not aware how reloading would be done.

[vishwambhar]

👍

[ddcprg]

👍adding a comment here just to track this

[muhammadkhilji]

+1. need this feature too:)

[SVLay]

+1.

[bgdnlp]

895 is related and probably the way forward for this request.

There are a number of similar-but-slightly-different implementations, it's clearly needed. It would be really good to have an official way of integrating EKS with Secrets Manager and Parameter store.

[jicowan]

Until there is a CSI driver, you can use this https://github.com/aws-samples/aws-secret-sidecar-injector

[jfgudk]

I think all these solutions are missing the mark and are overly complex, except https://github.com/godaddy/kubernetes-external-secrets .

If you have or use others' k8s applications that depend on Kubernetes secrets, these are not good solutions because they all require modification of the structure of your deployments.

IMO a solution that exposes/proxies/synchronizes secrets from Secrets Manager/SSM (or another backend source) as Kubernetes Secrets in a secure manner for consumption in a compatible manner to native Kubernetes Secrets should be the goal. This allows your applications to be portable.

[alaa]

I think all these solutions are missing the mark and are overly complex, except https://github.com/godaddy/kubernetes-external-secrets .

If you have or use others' k8s applications that depend on Kubernetes secrets, these are not good solutions because they all require modification of the structure of your deployments.

IMO a solution that exposes/proxies/synchronizes secrets from Secrets Manager/SSM (or another backend source) as Kubernetes Secrets in a secure manner for consumption in a compatible manner to native Kubernetes Secrets should be the goal. This allows your applications to be portable.

I agree with @jfgudk the beauty about cloud-native systems is the compatible interfaces among different components. So this should be always the goal. Also I have been using goDady's externa-secrets in a multi-tenant, multi-team environment across namesspaces and works like a charm. :+1:

[jicowan]

Thanks for the feedback @jfgudk and @alaa. As you say, the GoDaddy solution is making use of native Kubernetes Secrets. We created the Secret Sidecar Injector project in response to feedback from customers who told us they explicitly didn't want to use Kubernetes Secrets because they didn't feel they were a) secure enough [before KMS encryption of secrets] b) didn't provide fine grained access controls and c) didn't rotate automatically. With the Secret Sidecar Injector project you have to add a couple of annotations to your pod, but we think these changes are minimal.