HCP Vault Secret - VSO want to use secret from different namespaces

[mprochowski]

Describe the bug I have some applications that work in different namespaces on a cluster, and I want to provide passwords from HCP VS via VSO. I read the tutorial (which is very good 😄 ), but I got a small problem with the credentials secret. VSO requires from me to add a credentials secret in every namespace in which I used HCPVaultSecretApp object. For me is some kind of mismatch because VSO requires only one HCPAuth object on the cluster, but few identical secrets in every namespace that is used.

To Reproduce Steps to reproduce the behavior:

1. Deploy the application with the following yaml file with the following VSO custom resources.
2. See an error in the controller log

   manager 2023-10-23T21:00:46Z    ERROR    Get HCP Vault Secrets Client    {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVa
   ultSecretsApp", "HCPVaultSecretsApp": {"name":"web-application","namespace":"abc"}, "namespace": "abc", "name": "web-application", "reconcileID": "71442300-2edd-40da-8ace-79c6eae
   77f4e", "error": "failed to setup CredentialProvider, err=Secret \"vso-default-sp\" not found"}

Application deployment:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPAuth
metadata:
  name: default
  namespace: vault-secrets-operator-system
spec:
  organizationID: "..."
  projectID: "..."
  servicePrincipal:
    secretRef: vso-default-sp

---
apiVersion: v1
kind: Secret
metadata:
  name: vso-default-sp
  namespace: vault-secrets-operator-system
stringData:
  clientID: "..."
  clientSecret: "..."

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPVaultSecretsApp
metadata:
  name: web-application
  namespace: abc
spec:
  hcpAuthRef: "vault-secrets-operator-system/default"
  appName: abc-web-app
  destination:
    create: true
    name: abc-secret
  refreshAfter: 1h

Expected behavior HCPAuth should use credentials secret from the same namespace where it is.

Environment

• Kubernetes version: v1.27.6 (k3s)
• vault-secrets-operator version: v0.3.4

[mprochowski]

I checked the code and I think that the problem is caused by the controller, which passes an invalid namespace as a parameter.

1. https://github.com/hashicorp/vault-secrets-operator/blob/d2307002434ffd899f7982f9827f427dbb6295e6/controllers/hcpvaultsecretsapp_controller.go#L168-L171
2. https://github.com/hashicorp/vault-secrets-operator/blob/d2307002434ffd899f7982f9827f427dbb6295e6/internal/credentials/credentials.go#L58
3. https://github.com/hashicorp/vault-secrets-operator/blob/d2307002434ffd899f7982f9827f427dbb6295e6/internal/credentials/hcp/service_principal.go#L55-L63

Maybe the solution for this is something like this:

objKey := ctrlclient.ObjectKey{
        Namespace: l.authObj.Metadata.Namespace,
        Name:      l.authObj.Spec.ServicePrincipal.SecretRef,
    }
    secret, err := helpers.GetSecret(ctx, client, objKey)

However, this can be problematic due to backward compatibility.

[benashz]

Hi @mprochowski,

The current behavior is expected. The Vault Secrets Operator (VSO) will always access any credential provider e.g ServiceAccount, Secret, etc. in the K8s namespace of the referring VSO Secret CR. So in your case, that would be abc . In this way we have reduced the potential for a single K8s namespace from containing all of the authentication credential providers.

If you create your HCP SP creds K8s Secret in the abc namespace, VSO will then be able to authenticate to HCP and sync your secret.

Please let us know if that helps, or if you have any other questions.

Thanks,

Ben

[benashz]

One other thing, worth noting is that the default HCPAuth in VSO's namespace is special, in that it acts as a fallback in the case where no hcpAuthRef is specified on the HCPVaultSecretsApp CR. In the common use-case you would have your HCPAuth, and its credential source in the same namespace as your VSO Secret CR. Your app would only have permissions to access the destination K8s Secret via RBAC controls.

[mprochowski]

Hi @benashz,

When VSO uses Vault this solution makes sense to me, but not in the usage of HCP Vault Secrets. Referring to https://developer.hashicorp.com/hcp/docs/vault-secrets/structure-guide, I should create one project for env which has several apps with secrets. Moving on, I'm not able to create new credentials for specific applications, I can only do it for the project. So in this case, when I want to use HCP VS, I need to add the same HCPAuth and VSO Secrets CR in each namespace.

[mprochowski]

It is possible to use HCPAuth from eg. abc namespace in HCPVaultSecretApp object in namespace xyz. Maybe it's a good solution to add namespace parameter in HCPAuth that can refer to one secret in the whole cluster?

[2martens]

Stumbled across the same problem. HCPAuth contains a property allowedNamespaces which only makes sense if the same HCPAuth is linked to one secret of credentials that are for its project/organization. Currently I need to either deploy every application of one project in the same namespace which allows its pods to theoretically access all secrets, even unrelated ones to the application, OR I need to provide project-level service principal credentials in each of the application namespaces. That is also not least-privilege because now everyone with access to the application namespace can manually or otherwise retrieve the credentials and fetch all secrets from all applications.

If the VSO is not changed then application-level credentials are necessary in Vault Secrets. But that would provide the challenge of providing these credentials in a non-manual way without exposing them somewhere.

Currently, I install the credentials during cluster setup with terraform in the VSO namespace and the applications are installed later by Argo CD.

[mprochowski]

@benashz What we can do with this? Is it ok, if there will be namespace in HCPAuth that refer to secret?

I also think that this issue should be labeled as feature, not documentation 😄

[mprochowski]

Hi, is it possible to get some information about what's next with this problem? If you disagree, please let me know, so I can use another solution for my home lab, or if you agree how can I help.

[mprochowski]

@benashz ping 😄

[mprochowski]

@benashz ping 😄

[fergusean]

HCPAuth contains a property allowedNamespaces which only makes sense if the same HCPAuth is linked to one secret of credentials that are for its project/organization.

100% agree. I'm about to abandon HCP Vault Secrets in favor of 1Password Connect due to this issue 😅