Provide a way to use AWS Secrets Manager to get the sensitive configurations, like APIKEY and Database connection string

RoguePointer80 commented 4 years ago

Is your feature request related to a problem? Please describe.

I am trying to run BaGet in a Kubernetes cluster (AWS EKS to be precise). I run into the problem of providing the APIKEY and Database Connection String in a secure way. Since BaGet is running inside a container, I would need to provide environment variables; however putting the database password and api key in plaintext in the Kubernetes configmap is a terrible idea. Other environments might also have security concerns with exposing secrets in environment variables.

Describe the solution you'd like

I would like to provide in the environment variables and/or configuration files only the path to the AWS Secrets Manager ressource, like DB_ConnectionString_From_Aws_Secrets = /prod/database/connectionstring . During the Startup of the program, it would check if this key is set, and if so make an AWS client call to retrieve the secret value. Similar for API key. I like very much the modular approach in BaGet, and I'm sure Azure and GCP have similar services that could be leveraged.

Describe alternatives you've considered

I considered using Kubernetes secrets. But the problem is just moved elsewhere: where do I update the Kubernetes secret ? For instance AWS has all the mechanisms to automatically rotate the secrets, log and audit how is accessing the secrets and when. Trying to keep this in sync within Kubernetes can be a nightmare. Ref : https://github.com/aws/containers-roadmap/issues/168

Additional context

From what I've seen, the main things that need to be kept secret would be:

API key
Database connection string (containing both username and password) In case of secret rotation, the BaGet server needs to either reload them automatically (difficult, need to poll), or restart entirely (preferred, but how to know when to restart?). That's a bit beyond what this issue is about, since as a workaround I can ask the secret rotation lambda to restart the deployment.

RoguePointer80 commented 4 years ago

With some basic research I found 4 popular cloud services for secret management:

AWS Secrets Manager
Azure Key Vault
GCP Cloud KMS
Hashicorp Vault

Analogous to how we can use either AWS S3, Azure Storage or GCP storage, it would be great to be able to use any of these secret management services.

loic-sharma commented 4 years ago

Hello, thanks for opening this issue! BaGet is built on ASP.NET Core and supports the following configuration providers: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-3.1#configuration-providers

Unfortunately, ASP.NET Core does not have a provider for AWS Secrets Manager. Does AWS support Docker secrets by any chance? It's pretty easy to support those in an ASP.NET Core application, see: https://nvd.codes/post/use-docker-secrets-in-asp-net-core/

RoguePointer80 commented 4 years ago

EKS being a managed Kubernetes service, it doesn't use "Docker Secrets", which from what I understand (I might be wrong) is docker-compose specific. Instead it supports Kubernetes secrets, which is nearly the same thing. But as explained in the excellent blog post here,

If you're injecting environment variables at runtime (for example, into a docker container), then you need a process to do the injecting. That typically falls on your CI/CD server, which means it (the CI/CD server) needs to have knowledge (or know how to retrieve) all the required secrets for every application.

We essentially need a Configuration Provider like the Azure Key Vault, but for AWS Secrets Manager. After some basic research, that thing does exists 😄 https://github.com/Kralizek/AWSSecretsManagerConfigurationExtensions

Should I rename the issue or update its description? It should now be along the lines of : "Please integrate the AWS Secrets Manager configuration provider, so I can securely read my secrets in production." and of course, while we're at it, update the documentation to give examples of how that would work (same for Azure Key Vault).

loic-sharma / BaGet