[EKS] [request]: AWS Secrets Manager / SSM Parameter Store

[jaronsummers]

Tell us about your request It would be super great if things from secrets manager and/or parameter store could be exposed as secrets within Kubernetes.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We're in the middle of a somewhat stilted migration into Kubernetes, and there's not a ton of confidence that Kubernetes is the forever-future, and being able to manage secrets outside of kubernetes while accessing them natively would be very pleasant in terms of providing additional flexibility and making for a softer landing for devs.

Are you currently working around this issue? We are using native k8s secrets encrypted + stored in SCM, but that is probably not the future. I'm going to write + test a lambda that will push secrets into kubernetes, but I am unsure of how to manage namespacing and permissions, and how to do all of this securely. A lambda will also be complicated because it will either be one-way, have to maintain state to know which secrets have been removed and should be removed from kubernetes, or be the so-called single source of truth, and delete ALL secrets that aren't in secrets manager/ssm/whatever, which from a hygiene perspective sounds great, but from a dev experience perspective, much less so.

[jfgudk]

Understood. However, it also means you can't use many off the shelf deployments without rework.

[jqmichael]

Understood. However, it also means you can't use many off the shelf deployments without rework.

That's the downside.

With k8s secrets, the access is narrowed down to anyone who has read access to the secret in that namespace (also the secret is plain-text within apiserver/controller-manager and other master components, maybe Etcd if encryption isn't enabled).

With CSI volume based approach, the access is narrowed down only to people who has access to the pod.

[nalbam]

https://aws.amazon.com/ko/blogs/containers/aws-secrets-controller-poc/ AWS Secrets Controller PoC: integrating AWS Secrets Manager with Kubernetes by Jeremy Cowan, Amit Borulkar, Mahendra Revanasiddappa, and Satya Vajrapu | on 15 JUN 2020

[bjethwan]

Why not just support https://github.com/kubernetes-sigs/secrets-store-csi-driver, so we don't have to worry about securing secrets in etcd? I don't understand as why would aws be slow on supporting kubernetes-sigs/secrets-store-csi-driver when it makes perfect sense to integrate it with AWS Secres Manager. How come there's no public repo around initiation of that possible integration yet?

[jicowan]

Please review the minutes from the last community meeting, https://docs.google.com/document/d/1q74nboAg0GSPcom3kLWCIoWg43Qg3mr306KNL58f2hg/edit#heading=h.yvol7jx2oz82

[haad]

Hi maybe something like this which will load secrets to process ENV directly. https://github.com/lablabs/aws-secret-manager-loader

[matthewcummings]

@haad that's a good point. You can just have services bypass the native K8s constructs altogether and pull config data and/or secrets straight from Secrets Manager or Parameter Store when they start up.

[bgshacklett]

The problem with that is lack of portability. When services are designed to work with the built-in K8s constructs, they can (theoretically) be moved from platform to platform, or even deployed on multiple platforms, seamlessly. As soon as you start breaking out of those constructs, you now have to start supporting multiple versions of your pipeline, or even multiple versions of your container if your changes are significant enough.

If secrets manager is exposed, somehow, through the Kubernetes secrets facility, an application can be written to read from a secret and it has a guaranteed level of portability.

[wwentland]

Further to @bgshacklett comments, I would like to point out that some tools (e.g. helm-operator) rely on being able to load parts of their configuration from ConfigMaps or Secrets. If we were no longer able to utilise native K8s resources for exposing secrets to running workloads we would potentially have to adapt a wide variety of workloads/helm charts/kustomize overlays in an AWS specific manner.

[jfgudk]

Moreover this drives vendor lock in.

[bgdnlp]

@matthewcummings to clarify (I hope), check out kubernetes-external-secrets, also linked above. What it does is "import" secrets from AWS Secrets Manager as Kubernetes secrets. That way AWS Secrets are "translated" into native Kubernetes resources. Which provides portability. My bet is that that's what most people want from an AWS native solution, only officially supported and preferably preinstalled on EKS.

If you want your application to read secrets from AWS Secrets Manager directly, there's a myriad ways to do it, the most obvious being "use your favourite SDK to read secrets directly". That's not what I'm looking for, and I assume not what most of us want. Native K8s constructs is precisely what we'd like to have.

[matthewcummings]

@bgshacklett yes of course, I'm merely pointing out that you can just talk to Secrets Manager directly. This is perfectly fine if you're not worried about lock-in and/or using this pattern elsewhere e.g. with Lambda, ECS and anything else you might be running in AWS in addition to EKS clusters.

[DimaNevelev]

@haad that's a good point. You can just have services bypass the native K8s constructs altogether and pull config data and/or secrets straight from Secrets Manager or Parameter Store when they start up.

Pulling secrets into containers is not the only problem. K8s and other K8s extentions are realaing on secrets. Take a look on some use-cases:

• How do you solve ImagePullSecret?
• SVCAT is a K8s project that uses secrets in two flows:
  • Stores sensitive data in K8s secrets (the binding data that is returned from service brokers)
  • Broker registration with authentication

Eventually, we can solve everything using "bandage" fixes. But this is very bad practice, and hard to maintain.

[xzy0223]

hi, guys, any update on this feature? have a expected launch time?

[emcay]

Would love to know if this is something we can expect in the future. Would really like to have this.

[jwenz723]

Thanks for the feedback @jfgudk and @alaa. As you say, the GoDaddy solution is making use of native Kubernetes Secrets. We created the Secret Sidecar Injector project in response to feedback from customers who told us they explicitly didn't want to use Kubernetes Secrets because they didn't feel they were a) secure enough [before KMS encryption of secrets] b) didn't provide fine grained access controls and c) didn't rotate automatically. With the Secret Sidecar Injector project you have to add a couple of annotations to your pod, but we think these changes are minimal.

Sorry to revive an old comment. You state that customers have told you that kubernetes secrets are not secure enough. Couldn't they be considered secure if people properly use RBAC permissions to define access for users? In the clusters that I manage the majority of users have permissions to list secrets but do not have permissions to view the contents of the secrets.

Since kubernetes secrets have so many direct integrations to the other kubernetes resource types it seems like it would make the most sense to inject Secrets Manager values into a native kubernetes secret and maybe update the value of the kubernetes secret when the Secrets Manager secret is rotated.

Just my 2 cents.

[mikestef9]

Happy to share AWS Secrets Manager is now a supported provider for the Kubernetes Secrets Store CSI driver. The provider includes support for retrieving values from SSM Parameter Store as well.

More details:

• Launch Blog
• GitHub repository
• Documentation

[mveitas]

Is there an intention to enable this as an EKS addon?

[ysytnikmmg]

Does EKS Fargate support Kubernetes Secrets Store CSI driver? Or I should use mixed cluster ec2/fargate?

[yongzhang]

Does it support retrieving secrets by environment variables?

[Maxwell2022]

Does EKS Fargate support Kubernetes Secrets Store CSI driver

No, it's not supported because CSI Driver uses DeamonSet that is not supported by Fargate. The best option for Fargate is to use external-secret which uses a controller to fetch/sync secret from ASM and generates Kubernetes secrets that can then be used in your container as regular secrets.

Until AWS fixes this... Fargate is so far behind if you compare it with GCP that is fully managed