Ability to manage secrets across multiple kubernetes namespaces

[njohnstone2]

Is your feature request related to a problem? Please describe. The output kubernetes secret from a sync operation is always stored in the namespace that contains the VaultSecret* custom resource.

Theres two main problems with this:

1. I am unable to use the operator to manage secrets in scenarios where the secret is expected in a specific namespace.
2. a VaultConnection and VaultAuth resource needs to be created per kubernetes namespace that has vault managed secrets

Describe the solution you'd like It would be good if the Destination spec could support specifying the desired kubernetes namespace for the secret to be created in.

Describe alternatives you've considered Creating VaultConnection, VaultAuth and VaultSecret* custom resources all in the desired kubernetes namespace works as expected but this means that you end up with duplicate Connection/Auth custom resources spread across multiple kubernetes namespaces

[TheLonelyGhost]

Sounds like it's similarly adjacent to the problem posed in #128 where you want to distribute a standard secret across all namespaces. Is that the intended goal? Or do you just want a dedicated namespace to handle distributing secrets to other namespaces?

In a similar vein of lowering overhead to configure a new k8s namespace, I could see following the Cluster* pattern (i.e., ClusterRoleBinding vs Rolebinding, ClusterTask vs Task in Tekton) for resources like VaultConnection so it can be configured once and referenced by any kube namespace in the cluster.

[ameyp]

Or do you just want a dedicated namespace to handle distributing secrets to other namespaces?

That's the usecase I have in mind at least. I'd like to deploy the operator to the namespace containing my Vault ctluster, and have it be responsible for creating different secrets in different namespaces.

This also seems to follow from the defaultVaultConnection and defaultAuthMethod values in the Helm chart. Specify a default, and then that's responsible for mananging all secrets across the cluster.

[njohnstone2]

Or do you just want a dedicated namespace to handle distributing secrets to other namespaces?

This is my use case as well. Like @ameyp I also planned to use the default connection/auth from the helm values to manage my vault secrets.

In a similar vein of lowering overhead to configure a new k8s namespace, I could see following the Cluster* pattern (i.e., ClusterRoleBinding vs Rolebinding, ClusterTask vs Task in Tekton) for resources like VaultConnection so it can be configured once and referenced by any kube namespace in the cluster.

I would be happy with this solution. I think this would require the VaultSecret resource to exist in the destination namespace but that is fine for my use case.

Let me know if you disagree but I see two possible configuration patterns:

1. Have all the VaultSecret resources defined in a single namespace and the ability to specify a destination namespace
2. Define the VaultSecret resource in the target kubernetes namespace that references the vaultAuthRef in a separate namespace

[sebglon]

Using unique vaultAuthRef require to expose all your secret to the attached service account. and with this vaultAuth every namespace can request Secret all secret. for me using a unique vaultauth accross namespaces is a security breach.

[mrjebabli]

@njohnstone2 @sebglon @njohnstone2 @TheLonelyGhost so for the moment, we can only deploy secret on 1 namespace? I'm using this link to try vault-secrets-operator after installing of the plugin using this tutos (https://developer.hashicorp.com/vault/tutorials/kubernetes/vault-secrets-operator#the-vault-secrets-operator) helm install vault-secrets-operator hashicorp/vault-secrets-operator --version 0.1.0-beta -n vault --values vault/vault-operator-values.yaml

defaultAuthMethod:
enabled: true
# Vault namespace for the VaultAuthMethod CR
namespace: "app"
method: kubernetes
mount: kubernetes 

im only able to create secrets on the namespace app only, any idea how to add another namespace?

[njohnstone2]

I'm currently working around the limitation by duplicating the 'VaultConnection' and 'VaultAuth' resources to each namespace that needs to sync secrets

[mrjebabli]

thanks @njohnstone2 , so for the moment we can use the operator only for one namespace? we can't duplicate the 'VaultConnection' and 'VaultAuth' manually?

[togrulazizli]

It is possible to use single default VaultAuth and VaultConnection created by Helm chart in the namespace where the operator is installed. To be able to use the default VaultAuth from different namespace, the "vaultAuthRef:" should not be used in VaultSecret CRDs as pointed out in case #148 . Caveat is that a Service Account with the same name (the name which is defined in values file while deploying via Helm) must be created in all namespaces where VaultSecret CRDs will be used. Example: Vault Secrets Operator is being installed "vault-test" namespace via Helm with below values:

defaultVaultConnection:
  enabled: true
  address: "https://x.x.x.x"
  skipTLSVerify: true

defaultAuthMethod:
  enabled: true
  mount: dev-eks
  kubernetes:
    role: "dev-vault-secret-operator"
    serviceAccount: vault-secrets-operator-controller-manager

Now by default Helm chart creates Service Account for the operator named "vault-secrets-operator-controller-manager" so if you want you can use same Service Account for defaultAuthMethod or you can create and use another one. In above values, the helm created Service Account is being used. The importance of this Service Account name is that , Service Account with the same name depicted in the values file must be created in all namespaces where VaultSecret CRD will be deployed. Note: Since K8s 1.24 Service Account + its secret + token reviewer cluster role binding must be created by user. After the SA is deployed in required namespaces then VaultSecret CRD can be deployed without specifying "vaultAuthRef":

---
apiVersion: secrets.hashicorp.com/v1alpha1
kind: VaultStaticSecret
metadata:
  namespace: devops
  name: example-devops-static
spec:
  mount: sbx
  type: kv-v1
  name: toto-test
  refreshAfter: 60s
  destination:
    create: true
    name: toto-test-devops

Consclusion: example-devops-static VaultStaticSecret CRD will use VaultAuth default which is in Vault Namespace and default VaultAuth will look for the SA named vault-secrets-operator-controller-manager (or any other SA name which is defined in values file) in the Namespace where VaultStaticSecret CRD is deployed.

[allandegnan]

@togrulazizli - If you use the default service account like so:

---
defaultVaultConnection:
  enabled: true
  address: "http://vault.vault.svc.cluster.local:8200"
defaultAuthMethod:
  enabled: true
  kubernetes:
    role: ${vault-kube-auth-role}
    serviceAccount: default
    tokenAudiences:
    - vault

Then that already exists in every namespace.

The rest of my config is here: https://github.com/shutthegoatup/homelab and it appears to be working.

[nia-potato]

@allandegnan

Hey allan, i tried your method of deploying vso to a kubnenetes cluster that then connects back a external vault cluster, and the connections seems to be working fine, however when i try directly applying secrets such as:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: demo
  name: config
spec:
  mount: secret
  type: kv-v2
  name: myapp/config
  refreshAfter: 60s
  destination:
    create: true
    name: test-secret

I would get errors such as:

error: resource mapping not found for name: "config" namespace: "demo" from "secret2.yaml": no matches for kind "VaultStaticSecret" in version "secrets.hashicorp.com/v1beta1"
ensure CRDs are installed first

should we deploy secrets in a different way? I am assuming that since we already have defaultAuthMethod: block there is no need to re-apply something like the below since we are going through default service account auth.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: app
spec:
  method: kubernetes
  mount: test-dev
  kubernetes:
    role: vso
    serviceAccount: default

but it seems like on my end that if i dont apply VaultAuth, when applying secrets even when getting Connection Accepted in the vso logs i would get ensure CRDs are installed first

[allandegnan]

@allandegnan

Hey allan, i tried your method of deploying vso to a kubnenetes cluster that then connects back a external vault cluster, and the connections seems to be working fine, however when i try directly applying secrets such as:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: demo
  name: config
spec:
  mount: secret
  type: kv-v2
  name: myapp/config
  refreshAfter: 60s
  destination:
    create: true
    name: test-secret

I would get errors such as:

error: resource mapping not found for name: "config" namespace: "demo" from "secret2.yaml": no matches for kind "VaultStaticSecret" in version "secrets.hashicorp.com/v1beta1"
ensure CRDs are installed first

should we deploy secrets in a different way? I am assuming that since we already have defaultAuthMethod: block there is no need to re-apply something like the below since we are going through default service account auth.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: app
spec:
  method: kubernetes
  mount: test-dev
  kubernetes:
    role: vso
    serviceAccount: default

but it seems like on my end that if i dont apply VaultAuth, when applying secrets even when getting Connection Accepted in the vso logs i would get ensure CRDs are installed first

Assuming the CRD message is a red herring, I think your path is probably wrong.

My mount point is "kvv2" and my path is everything after that.

- apiVersion: secrets.hashicorp.com/v1beta1
  kind: VaultStaticSecret
  metadata:
    annotations:
      meta.helm.sh/release-name: arc-github-app
      meta.helm.sh/release-namespace: actions-runner-controller
    creationTimestamp: "2023-07-02T13:42:32Z"
    generation: 1
    labels:
      app: raw
      app.kubernetes.io/managed-by: Helm
      chart: raw-v0.3.2
      heritage: Helm
      release: arc-github-app
    name: arc-github-app
    namespace: actions-runner-controller
    resourceVersion: "8008422"
    uid: e1d23224-7df5-40a1-9e73-648e32cb2868
  spec:
    destination:
      create: true
      name: arc-github-app
    hmacSecretData: true
    mount: kvv2
    path: arc-github-app
    refreshAfter: 60s
    type: kv-v2

Here's a screenshot of my vault https://imgur.com/a/7WZ7dta

Hope that helps.

[ChristianCiach]

I am surprised that this issue has been closed as completed. It's still not possible to use a single, global ServiceAccount for VaultStaticSecrets across all namespaces of the cluster.

Even when using this VaultAuth, you still need a separate vault-default ServiceAccount in each namespace:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: default
spec:
  vaultConnectionRef: default
  allowedNamespaces: ['*']
  method: jwt
  mount: jwt-external-services
  jwt:
    role: vso-default
    serviceAccount: vault-default
    audiences:
      - vault  # must match the `bound_audiences` of the vault role 

EDIT: Please disregard that. While this is technically true, I just found out that bound_subject of the role definition is an optional attribute. I can just use bound_audiences and use the default ServiceAccount that exists by default in every namespace.