search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.13k stars 4.21k forks source link

Issues with Vault plugins intergation #14129

Closed SachinMaharana closed 2 years ago

SachinMaharana commented 2 years ago

Describe the bug I am having issues relating to setup plugin for vault. While vault works fine and i could get the secrets as injector with tls enabled, but setting up plugins is hitting a roadblock. While writing the config file for plugin, i get the error.

To Reproduce

vault write sys/plugins/catalog/secret/op-connect \
sha_256="$(sha256sum /vault/data/plugins/op-connect | cut -d " " -f1)" \
command="op-connect -ca-cert=/vault/userconfig/vault-server-tls/vault.ca"

vault secrets enable --plugin-name='op-connect' --path="op" plugin

// I get a issue here
vault write op/config @op-connect-config.json

secrets.op-connect.op-connect_96388717.op-connect.op-connect: plugin tls init: error="error during token unwrap request: Put \"https://10.4.1.8:8200/v1/sys/wrapping/unwrap\": x509: certificate is valid for 127.0.0.1, 34.xx.xx.xx, not 10.4.1.8" timestamp=2022-02-17T18:06:44.542Z
2022-02-17T18:06:44.547Z [INFO]  http: TLS handshake error from 10.4.1.8:43720: remote error: tls: bad certificate

Not sure why the API_ADDR is the pod ip, which is not present in the SAN of the csr as seen below

My CSR config is 

cat <<EOF >${TMPDIR}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_INTERNAL_SVC}
DNS.2 = *.${NAMESPACE}.svc.cluster.local
DNS.3 = *.${VAULT_INTERNAL_SVC}.${NAMESPACE}.svc.cluster.local
DNS.4 = vault.${NAMESPACE}.svc.cluster.local
DNS.5 = vault-agent-injector-svc
DNS.6 = vault-agent-injector-svc.vault
DNS.7 = vault-agent-injector-svc.vault.svc
DNS.8 = vault-agent-injector-svc.vault.svc.cluster.local
DNS.9 = vault
DNS.10 = vault.${NAMESPACE}
DNS.11 = vault.${NAMESPACE}.svc
IP.1 = 127.0.0.1
IP.2 = 34.xx.xx.xx
EOF

Any hint would be of great help. Thanks

Steps to reproduce the behavior:

  1. Run vault write ...
  2. Run vault login....
  3. See error

Expected behavior Expected the plugin to work

Environment:

* Vault CLI Version (retrieve with `vault version`):

Vault v1.9.3 (7dbdd57243a0d8d9d9e07cd01eb657369f8e1b8a)

* Server Operating System/Architecture:
Kubernetes with vault helm chart, HA, Integrated Storage, TLS enabled
Vault server configuration file(s):

```hcl
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  agentImage:
    repository: "hashicorp/vault"
    tag: "1.9.3"

csi:
  enabled: true
  volumes:
    - name: vault-server-tls
      secret:
        secretName: vault-server-tls
  volumeMounts:
    - name: vault-server-tls
      mountPath: /mnt/tls
      readOnly: true

server:
  extraInitContainers:
    - name: op-connect
      image: "alpine"
      command: [sh, -c]
      args:
        - >
          cd /tmp;
          wget -O op-connect.zip
          https://github.com/1Password/vault-plugin-secrets-onepassword/releases/download/v1.0.0/vault-plugin-secrets-onepassword_1.0.0_linux_amd64.zip;
          unzip op-connect.zip;
          mv vault-plugin-secrets-onepassword_v1.0.0 /vault/data/plugins/op-connect;
          chmod +x /vault/data/plugins/op-connect;
          touch op-connect-config.json;
          mv op-connect-config.json /vault/data/plugins;
      volumeMounts:
        - name: plugins
          mountPath: /vault/data/plugins
  volumes:
    - name: plugins
      emptyDir: {}
  volumeMounts:
    - mountPath: /vault/data/plugins
      name: plugins
  image:
    repository: "hashicorp/vault"
    tag: "1.9.3"

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault-server-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-server-tls/vault.key
    # GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/vault-gcs/vault_gcs_key.json

  extraVolumes:
    - type: secret
      name: vault-server-tls
    # - type: secret
    #   name: vault-gcs
    #   path: "/vault/userconfig"

  auditStorage:
    enabled: true
    mountPath: "/vault/audit"
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false

      config: |
        ui = true
        api_addr = "http://POD_IP:8200"
        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"

          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
        }
        service_registration "kubernetes" {}
        disable_mlock = true
        plugin_directory = "/vault/data/plugins"

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 3
            server_stabilization_time = "10s"
          }

        }
ui:
  enabled: true
  serviceType: "NodePort"
  serviceNodePort: 30082
  externalPort: 8200

Additional context Add any other context about the problem here.

ccapurso commented 2 years ago

Hi, @SachinMaharana. Thank you for your engagement! Our use of GitHub issues is meant to enable the Vault community the to submit feature requests and bug reports. The Vault Discuss forum is the best venue to seek feedback on general usage issues. It appears that you have submitted a similar issue there already. As such, I'm going to close this issue.

phishbandana commented 1 year ago

Can we open this back up or submit this to support? The question here isn't answered: similar issue

maxb commented 1 year ago

The problem reported here relates in part to the issue #15070 that I opened.

Specifically, Vault plugins connect back to the api_addr of the Vault that started them, as part of their initialization.

This is an obscure detail that I haven't found documented anywhere, and needed to learn from the source code.

Therefore what is going wrong here, is that the user is attempting to run a Vault configured with an SSL certificate that is not valid for the hostname/IP address in its configured api_addr.

My issue #15070 goes into further detail about why I think plugins should not use api_addr as part of their initialization.