hashicorp / vault-helm

Helm chart to install Vault and other associated components.
Mozilla Public License 2.0
1.05k stars 868 forks source link

Configuring vault ha with raft and ingress #1020

Closed Mdumala closed 1 month ago

Mdumala commented 2 months ago

Hello

I'm trying to create vault HA with raft(storage) and use ingress

I'm doing it by flux

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: vault
  namespace: vault
spec:
  interval: 30m
  chart:
    spec:
      chart: vault
      version: 0.27.0
      sourceRef:
        kind: HelmRepository
        name: hashicorp
      interval: 30m
  values:
    global:
      tlsDisable: false
      namespace: vault
    injector: 
      enabled: false
    server:
      extraEnvironmentVars:
        VAULT_CACERT: /vault/userconfig/vault-server-tls/ca.crt
        VAULT_TLSCERT: /vault/userconfig/vault-server-tls/vault.crt
        VAULT_TLSKEY: /vault/userconfig/vault-server-tls/vault.key
        VAULT_ADDR: "https://vaultlab.test.pl:8200"
      volumes:
        - name: userconfig-vault-server-tls
          secret:
           defaultMode: 420
           secretName: vault-server-tls
      volumeMounts:
        - mountPath: /vault/userconfig/vault-server-tls
          name: userconfig-vault-server-tls
          readOnly: true
      logLevel: "info"
      ingress:
        enabled: true
        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        ingressClassName: nginx
        pathType: Prefix
        activeService: true
        hosts:
          - host: vaultlab.test.pl
        tls: 
           - secretName: vault-server-tls
             hosts:
               - vaultlab.test.pl
        nodeSelector: 
          node-role.kubernetes.io/worker: worker
      dataStorage:
        enabled: true
        size: 5Gi
        mountPath: "/vault/data"
        storageClass: nfs-client-1
        accessMode: ReadWriteOnce
        labels: 
          app: vault
      auditStorage: 
        enabled: true
        size: 5Gi
        mountPath: "/vault/audit/"
        storageClass: nfs-client-1
        accessMode: ReadWriteOnce
        labels: 
          app: vault
      ha:
        enabled: true
        replicas: 3
        #apiaddr: "https://vaultlab.test.pl:8200"
        #clusterAddr: "https://vaultlab.test.pl:8200"
        raft: 
          enabled: true
          setNodeId: true
          config: |
            ui = true
            listener "tcp" {
               tls_disable = "false"
               tls_client_ca_file = "/vault/userconfig/vault-server-tls/ca.crt"
               tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
               tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
             }
             storage "raft" {
             path = "/vault/data"
               retry_join {
               leader_api_addr = "https://vault-0.vault-internal:8200"
               leader_ca_cert_file = "/vault/userconfig/vault-server-tls/ca.crt"
               leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
               leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
             }
               retry_join {
               leader_api_addr = "https://vault-1.vault-internal:8200"
               leader_ca_cert_file = "/vault/userconfig/vault-server-tls/ca.crt"
               leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
               leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
             }
               retry_join {
               leader_api_addr = "https://vault-2.vault-internal:8200"
               leader_ca_cert_file = "/vault/userconfig/vault-server-tls/ca.crt"
               leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
               leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
            }
            }
             service_registration "kubernetes" {}

The problem is that i cannot initialize vault (vault operator init) beacue its tries to reach api by address :

Get "https://vaultlab.test.pl:8200/v1/sys/seal-status": dial tcp 10.250.xxx.xxx:8200: connect: no route to host which is clear for me beacuse service which is under ingress expect label vault-active=true when the pods have label vault-active=false (unit they are initialized i think)

How i can bypass this? Maybe i should create ingress after initialized vault? Or maybe i'm doing something wrong with this setup.

I'm opening for your insight.

Best Regards

Mdumala commented 1 month ago

Well it works now. Turns out that it was enough to "init" vault locally using --address parameter (with IP parameter in certificate for localhost).