Unable to inject Vault CA into Vault Helm chart, injector complains w/self-signed cert

[PerilousApricot]

Describe the bug When deploying Vault Agent with an external vault instance, there is no way to tell the webhook the vault CA, so all accesses complain about a self-signed CA

To Reproduce Steps to reproduce the behavior:

1. Generate certs from your internal CA, provide them to your bare-metal vault instance
2. Install chart w/external address option
3. Deploy a k8s service with annotations requesting secrets from vault
4. init-sidecar fails TLS validation because the sidecar doesn't know of the CA root

Expected behavior Our non-k8s-deployed infra trusts the CA roots because we inject our CA into /etc/pki -- there should be a way to tell the agent which CAs are worthwhile

Environment

• Kubernetes version: 0.23.7
• vault-helm version: 0.21.0

Chart values:

I've tried to make cert-bot generate a (dummy) cert, manually change the injector pod to inject the cert as a secret, then set the VAULT_CACERT variable to the path of our CA, but the agent doesn't accept it

[alekc]

EDIT: nvm, just saw this discussion: https://github.com/hashicorp/vault-k8s/issues/19

TLDR: it looks like you have to create a copy of a ca in a secret in every ns you want to use, and then use following annotations to inject it

vault.hashicorp.com/tls-secret:  “client-vault-auth”
vault.hashicorp.com/ca-cert: “/vault/tls/client.ca”

---

I think I am hitting the same issue but with the vault and injectors being in the same cluster.

Certs are created by terraform (should have the same result as self signed CA)

TF CA

```hcl // create self signed ca & certs for the vault installation resource "tls_private_key" "vault_ca" { algorithm = "RSA" rsa_bits = 4096 } resource "tls_private_key" "vault_rsa" { algorithm = "RSA" rsa_bits = 4096 } resource "tls_self_signed_cert" "vault_ca" { private_key_pem = tls_private_key.vault_ca.private_key_pem subject { common_name = "Vault Root CA" organization = "Acme Inc" organizational_unit = "Development" street_address = ["1234 Main Street"] locality = "Beverly Hills" province = "CA" country = "USA" postal_code = "90210" } # 175200 = 20 years validity_period_hours = 175200 allowed_uses = [ "cert_signing", "crl_signing" ] is_ca_certificate = true } resource "tls_cert_request" "vault" { private_key_pem = tls_private_key.vault_rsa.private_key_pem subject { common_name = var.app_vault_service_name organization = "ACME Examples, Inc" } ip_addresses = [ "127.0.0.1" ] dns_names = [ var.app_vault_service_name, "${var.app_vault_service_name}-active", "${var.app_vault_service_name}-0.vault-internal", "${var.app_vault_service_name}-1.vault-internal", "${var.app_vault_service_name}-2.vault-internal", "${var.app_vault_service_name}.${var.app_vault_namespace}", "${var.app_vault_service_name}.${var.app_vault_namespace}.svc", "${var.app_vault_service_name}.${var.app_vault_namespace}.svc.cluster.local", ] } resource "tls_locally_signed_cert" "vault" { cert_request_pem = tls_cert_request.vault.cert_request_pem ca_private_key_pem = tls_private_key.vault_ca.private_key_pem ca_cert_pem = tls_self_signed_cert.vault_ca.cert_pem validity_period_hours = 87600 # 24*365*10 allowed_uses = [ "key_encipherment", "digital_signature", "server_auth", ] } resource "kubernetes_secret" "vault_cert_secret" { count = var.app_vault_enable ? 1 : 0 depends_on = [kubernetes_namespace.vault_ns] metadata { name = "vault-server-tls" namespace = var.app_vault_namespace } data = { "vault.key" = tls_private_key.vault_rsa.private_key_pem "vault.crt" = tls_locally_signed_cert.vault.cert_pem "vault.ca" = tls_self_signed_cert.vault_ca.cert_pem } } ```

Vault config

```yaml fullnameOverride: "vault" global: enabled: true tlsDisable: false injector: enabled: true resources: requests: memory: 256Mi cpu: 250m server: extraVolumes: - type: secret name: ${kubernetes_secret.gcp-vault-auth[0].metadata.0.name} resources: requests: memory: 256Mi cpu: 500m ingress: enabled: true annotations: nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" hosts: - host: vault.alekc.dev paths: ["/"] affinity: "" # For HA configuration and because we need to manually init the vault, # we need to define custom readiness/liveness Probe settings readinessProbe: enabled: true path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204" livenessProbe: enabled: true path: "/v1/sys/health?standbyok=true" initialDelaySeconds: 60 # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be # used to include variables required for auto-unseal. extraEnvironmentVars: VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca GOOGLE_CREDENTIALS: /vault/userconfig/gcp-auth/credentials volumes: - name: userconfig-vault-server-tls secret: secretName: ${kubernetes_secret.vault_cert_secret[0].metadata.0.name} defaultMode: 420 volumeMounts: - name: userconfig-vault-server-tls readOnly: true mountPath: /vault/userconfig/vault-server-tls # auditStorage: # enabled: true dataStorage: enabled: true size: 5Gi #10 storageClass: ${kubernetes_storage_class.longhorn-not-replicated.metadata.0.name} serviceAccount: create: true ha: enabled: true replicas: 3 raft: enabled: true setNodeId: true # language=hcl config: | ui = true listener "tcp" { address = "[::]:8200" cluster_address = "[::]:8201" tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" tls_key_file = "/vault/userconfig/vault-server-tls/vault.key" tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca" } storage "raft" { path = "/vault/data" retry_join { leader_api_addr = "https://vault-0.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } retry_join { leader_api_addr = "https://vault-1.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } retry_join { leader_api_addr = "https://vault-2.vault-internal:8200" leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca" leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key" } autopilot { cleanup_dead_servers = "true" last_contact_threshold = "200ms" last_contact_failure_threshold = "10m" max_trailing_logs = 250000 min_quorum = 5 server_stabilization_time = "10s" } } service_registration "kubernetes" {} seal "gcpckms" { project = "vault-360115" region = "eur5" key_ring = "home-vault" crypto_key = "vault" } service_registration "kubernetes" {} standalone: enabled: false } ```

Pki backend is configured.

Once I start the installation (consul), every pod is in the pod initializing state, with vault-agent-init reporting 2022-08-23T19:17:36.670Z [ERROR] auth.handler: error authenticating: error="Put \"https://vault.vault.svc:8200/v1/auth/kubernetes/login\": x509: certificate signed by unknown authority" backoff=4m27.01s

[tomhjp]

Thanks for the input on this, should be better fixed in #507 now.