search

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
471 stars 101 forks source link

VSO Proxy Support #638

Open hinrichd opened 8 months ago

hinrichd commented 8 months ago

Describe the bug When using vso on an onpremise cluster behind a proxy, vso do not use the proxy varibales from extraEnv to lookup and connect to Hashicorp Vault Secrets. Connection Test fails due to lookup on KubenDNS and not using proxy.

To Reproduce Steps to reproduce the behavior:

  1. Deploy VSO with extraEnv in helm values: helm install vault-secrets-operator hashicorp/vault-secrets-operator -f values.yaml 
    controller:
manager:
extraEnv:
  - name: http_proxy
    value: "http://PROXYURL:PORT"
  - name: https_proxy
    value: "http://PROXYURL:PORT"
  - name: no_proxy
    value: "127.0.0.1,localhost,.local,.svc,.svc.cluster.local"
  2. Check VSO Pod Describition if proxy env is correctly applied: 
    Environment:
  OPERATOR_POD_NAME:          vault-secrets-operator-controller-manager-58c4dc5674-dbp75 (v1:metadata.name)
  OPERATOR_POD_UID:            (v1:metadata.uid)
  KUBERNETES_CLUSTER_DOMAIN:  cluster.local
  http_proxy:                 http://PROXYURL:PORT
  https_proxy:                http://PROXYURL:PORT
  no_proxy:                   127.0.0.1,localhost,.local,.svc,.svc.cluster.local
  3. Apply Secret with SP, HCPAuth and HCPVaultSecretsApp for Test-Projekt.
  4. Check VSO Pod Logs

"error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X.X53: server misbehaving" Lookup using internal kubedns and not using proxy, so it will fail. 

2024-03-05T07:48:52Z    INFO    Starting workers        {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "worker count": 100}
2024-03-05T07:48:52Z    ERROR   Validation failed       {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "HCPAuth": {"name":"test-projekt","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test-projekt", "reconcileID": "4c8eb156-e77a-4e17-82ed-9431410eed7d", "error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X.X:53: server misbehaving"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPAuthReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpauth_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:52Z    ERROR   Validation failed       {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "HCPAuth": {"name":"default","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "default", "reconcileID": "9c399810-b5da-4109-95b5-5955acc9be92", "error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X:X:53: server misbehaving"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPAuthReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpauth_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:53Z    ERROR   Get App Secret  {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "HCPVaultSecretsApp": {"name":"test","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test", "reconcileID": "73167846-9b0f-4657-9116-49fdf19bf777", "appName": "test", "error": "[GET /secrets/2023-06-13/organizations/{location.organization_id}/projects/{location.project_id}/apps/{app_name}/open][403] OpenAppSecrets default  &{Code:7 Details:[] Message:}"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPVaultSecretsAppReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpvaultsecretsapp_controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:59Z    ERROR   Get App Secret  {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "HCPVaultSecretsApp": {"name":"test","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test", "reconcileID": "dfc56edc-d055-47e4-ab26-b5069b4da7ac", "appName": "test", "error": "[GET /secrets/2023-06-13/organizations/{location.organization_id}/projects/{location.project_id}/apps/{app_name}/open][403] OpenAppSecrets default  &{Code:7 Details:[] Message:}"}
  1. Deployed CURL Pod in same namespace with proxy env and and manually curled secrets from Vault Secrets to verify connection is working on cluster behind a proxy (working). Curl using lowercase http_proxy env, so also tested helm extraEnv with lower case proxy variable.

Environment

Thank You for your help.

benashz commented 8 months ago

Hi @hinrichd,

Thank you for the detailed report. I see two issues here.

  1. The connection check on the HCPAuth resource is failing. This check is used for information purposes only, and should not affect your HCPVaultSecretsApp being synced to K8s.
  2. From the HCPVaultSecretsApp logs, it seems that VSO is getting a 403 (permission denied error) when trying to fetch the Vault Secrets app. That seems like it may be related to the HCP client credentials configured on HCPAuth resource. Can you confirm that the credentials that VSO is using have the right permissions to access the HCP Vault Secrets app in question? The 403 error could also be coming from your HTTPS proxy, can you confirm that proxy URL has the correct credentials if authentication is being enforced on the proxy side?

I took a closer look at the code and did some internal testing, and it seems setting HTTPS_PROXY (lower or upper case) is being honoured. So more than likely the issue is related to the permission settings on the HCP Vault Secrets side not matching what you have in your service principal credentials. You will want to verify those credentials match the OrganizationID/ProjectID configured on your HCPAuth.

Also, just be aware that the approach you are taking by setting the proxy globally for VSO, may cause some unexpected behaviour. Ideally, VSO would support granular proxy configuration on the HCPAuth or VaultConnection custom resources. We will probably add support for that use case in a future VSO release.

Ben

hinrichd commented 8 months ago

Hi @benashz

Thank you for pointing out some details to my related problem. I could solve it now by changing the sp permissions. Sounds good to have some granular proxy config on the HCP Auth in the future release.

THX Hinrich