RFD - Vault for Deployment and Dynamic User Secrets

[costrouc]

Status	Open for comments 💬
Author(s)	costrouc
Date Created	14-02-2022
Date Last updated	14-02-2022
Decision deadline	14-02-2022

Vault for Deployment and Dynamic User Secrets

Summary

Have spent around 2 days familiarizing myself with Vault and trying it out by using Hashicorp's managed vault deployment. We have the features that we would need to allow:

• storing deployment secrets
• allowing users/groups to create their own secrets which they can then update/delete and share with other services all with a rich permissions model

User benefit

There are two users I have in mind this this proposal end users e.g. regular users/developers on Nebari and devops/it sysadmins managing the deployment of Nebari. This proposal would satisfy both of these.

Design Proposal

Implementation

• would be nice to have monitoring (prometheus/grafana) in a separate stage since vault can export metrics to prometheus (optional)
• 2 new stages before 07-kubernetes-services
  • vault deployment via helm
  • configure vault
    • after this is done we store all the secrets created during the deployment in vault using kubernetes authentication via a service account we created during the deployment.

Notice user does not have to store/remember any secrets!

How would we configure vault:

• auth provider:
  • kubernetes auth using kubernetes service accounts (for deployments and services)
  • oidc auth using keycloak (for users)
• policies would be:
  • created for users/groups to have paths e.g. users/<username>/* and group/<group>/* to write arbitrary secrets
  • created for services to that the given service only has access to those specific secrets
    • how to mount the secrets. vault has several options (sidecar looks the most promising since it allows for dynamically updating secrets) https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar

# patch-basic-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
        vault.hashicorp.com/role: "myapp"

Kubernetes service accounts are at the heart of this. Would would assign identities to users/services via attaching a service account e.g. <namespace>/service-<service-name> or <namespace>/user-<username>

Alternatives or approaches considered (if any)

There is currently a proposal for using SOPS for secret management https://github.com/nebari-dev/governance/issues/29.

• SOPS to me has several downsides
  • yet another place to store the secrets
  • managing private keys to encrypt/decrypt secrets
  • support for multiple public/private keys?
  • does not address the regular user use case of wanting dynamic secrets on the cluster
  • additional things to manage in the nebari github repo
  • dangers of commiting secrets to repo unencrypted
  • Advantages to me:
  • significantly simpler to deploy

Best practices

User impact

Unresolved questions

Do we use separate namespaces for users?

[iameskild]

I agree that Vault provides a suite of additional features and benefits over SOPS. As I mentioned elsewhere, Vault is to secret management what Keycloak is to user management, a professional tool.

If we're going to spend the time implementing one or the other, I would be in favor of implementing Vault.

[dharhas]

So I like the end user experience of naas.ai

see: https://docs.naas.ai/

Add Secret Keys

Copy in production any secret key :

naas.secret.add(name="API_NAME", secret="API_KEY")

Remove the previous line and get your secret key with :

naas.secret.get(name="MY_API_KEY")

This allows you to push your notebook in production without sensitive data getting exposed.

No idea how it is implemented/

[viniciusdc]

@costrouc Are we moving into this direction yet?