ha mode fails to check status

[MrAmbiG]

kubectl exec -it vault-0 -- vault status Error checking seal status: Get http://localhost:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: connect: connection refused.

'kubectl exec -it vault-0 -- vault operator init -n 1 -t 1 Error initializing: Put http://127.0.0.1:8200/v1/sys/init: dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 2'

Official documentation i followed: https://www.hashicorp.com/blog/announcing-the-vault-helm-chart/

1. deployed consul helm chart
2. deployed vault helm chart with in ha mode The above 2 errors is something i get only if i deploy in ha mode. If i deploy in standalone (default) mode then all is well. Unfortunately since it is deployed now using helm, creating a replicaset for this statefulset is out of question. Please fix this.

[udaymadu]

is bug has any timeframe

[smartpierre]

I got the same issue, but after 2 to 3 minutes, the pods start properly 🤷‍♂️

[ko5tas]

Deployed to EKS as per instruction in https://www.vaultproject.io/docs/platform/k8s/helm/run but when it came to initializing the cluster I also got the same error.

I did the kubectl port-forward vault-0 8200:8200 but did not work...

$ kubectl exec -it vault-0 -- vault status Error checking seal status: Get http://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 1

[byronmansfield]

I am also seeing this issue. Has anyone figured out what the issue is?

[Mr-Howard-Roark]

I am also running into this out of nowhere. Using the exact same Docker Image from vault:1.5.4 and exact same configuration (ie- values.yaml) I never ran into this in other setups, but now I am tonight.

[balajimejari]

I got the same issue did any one has solution for that

[balajimejari]

I got the same issue . $ oc exec -it vault-0 vault status Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 1 when I look into pod logs I can see below 2020-11-04T12:23:17.945Z [WARN] storage migration check error: error="Get "http://10.160.225.18:8500/v1/kv/vault/core/migration": dial tcp 10.160.225.18:8500: connect: connection refused" So what I understood is , 10.160.225.18(HOST_IP) is my worked node where consul server pod is running , vault is not connecting to consul server with HOST_IP with 8500 port number , and below is my values .yaml storage "consul" { path = "vault/" address = "HOST_IP:8500"

what the work around I did was , I Changed HOST_IP:8500 to my consul SERVICE_IP , hence it is headless service , service ip not generated so I have given my consul service name , in my case my consul service name is "consul-server" balaji@DESKTOP-O8C6N39:~/vault$ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE consul-dns ClusterIP 172.21.33.6 53/TCP,53/UDP 2d consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP

values.yaml

storage "consul" { path = "vault" address = "consul-server:8500"

Then vault was deployed and working fine SO finally my vault HA with backend consul storage is working perfectly .

[r3mattia]

I have a similar problem when deploying Vault in HA mode with Consul as the storage backend configured to TLS as well with a self-signed CA. I simply generate the certs with the consul binary / CLI.

I was able to deploy Vault without TLS on top of the Consul backend, but I am continuously encountering the following connection refused issue when I deploy Vault with TLS enabled:

/ $ vault status
Error checking seal status: Get "https://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
/ $ vault operator init
Error initializing: Put "https://127.0.0.1:8200/v1/sys/init": dial tcp 127.0.0.1:8200: connect: connection refused
/ $

My consul helm values file looks as follows:

global:
  enabled: true
  domain: consul
  name: consul
  datacenter: consul-primary
  tls:
    # TLS must be enabled for federation in Kubernetes.
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
  # Consul CA cert
  caCert:
    # The name of the Kubernetes secret.
    secretName: consul-ca-cert
    # The key of the Kubernetes secret.
    secretKey: tls.crt

  caKey:
    # The name of the Kubernetes secret.
    secretName: consul-ca-key
    # The key of the Kubernetes secret.
    secretKey: tls.key

server:
  replicas: 3
  bootstrapExpect: 3
  # Consul server certs
  extraVolumes:
    - type: secret
      name: consul-server-cert
      load: true

ui:
  enabled: true
  service:
    enabled: true
    type: ClusterIP

My Vault values file looks as follows:

global:
  enabled: true
  tlsDisable: false

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  extraVolumes:
  - type: secret
    name: vault-server-cert
  - type: secret
    name: consul-server-cert
  - type: secret
    name: consul-ca-cert

  ha:
    enabled: true
    replicas: 3 
    config: |
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-cert/tls.crt"
        tls_key_file  = "/vault/userconfig/vault-server-cert/tls.key"
      }

      storage "consul" {
        path = "vault"
        address = "$HOST_IP:8501"
        tls_key_file = "/vault/userconfig/consul-server-cert/tls.key"
        tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt"
        tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt"
      }

      service_registration "consul" {
        address = "$HOST_IP:8501"
        tls_key_file = "/vault/userconfig/consul-server-cert/tls.key"
        tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt"
        tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt"
      }

ui:
  enabled: true
  serviceType: "ClusterIP"
  externalPort: 8200

I have been stuck on this for days now given that none of the docs actually explains the process accurately... Please fix this.

[balajimejari]

Hi Vault is not able to communicating to backend consul host with that given port , so instead of HOST_IP:8501 , you can give your consul service ip or consul service name under the storage section in vault values.yaml file

storage "consul" { path = "vault" address = "your consul service ip or consul service name:8500" ( in my case I gave my consul service name ) tls_key_file = "/vault/userconfig/consul-server-cert/tls.key" tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt" tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt" }

Hope this will resolve your issue ..

[MuhammadJamee]

Just start consul along with it. The error will be resolved. ref:https://learn.hashicorp.com/tutorials/consul/kubernetes-deployment-guide?in=consul/kubernetes

[muhammadmuhlas]

worked using this solution! vault-values.yaml

server:
  enabled: true
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "consul" {
        path = "vault/"
        address = "consul-server:8500"
      }
      service_registration "kubernetes" {}

kubectl exec -it vault-0 -n vault -- vault status

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.9.0
Storage Type       consul
HA Enabled         true

[jerrac]

Thanks @muhammadmuhlas ! I used a namespace when installing the Consul helm chart, so the name of my Consul service was different than what the Vault helm chart defaults to. Your config showed me how to update that.

[vieenodp]

Vault HA using Helm on OpenShift. Getting this error.

Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

[rnpaiva]

news ? I have the same issue

[swenson]

Hi,

Issues like this are often caused by Vault not being fully up and running (still initializing) or storage not being fully up and running. I'd double-check that each pod is healthy for your storage and Vault cluster, and check the logs on each to see if they are having some trouble initializing. If that doesn't work, I'd try again by following one of the tutorials.

If you have further issues, you might have better luck on the Vault discussion board.

Going to close for now as this doesn't appear to be a bug

[christiancorbin]

I figured this one out for my cluster, its the CNI not supporting HostPort by default per: https://developer.hashicorp.com/consul/docs/troubleshoot/common-errors At the bottom of the page: "If you are still unable to connect and see i/o timeout or connection refused errors when connecting to the Consul client on the Kubernetes worker, this could be because the CNI (Container Networking Interface) does not support the use of hostPort."

I was using kube-router for CNI and it doesn't support HostPort by default. The error says connection is refused by 127.0.0.1:8200 but the issue is actually with connecting to HOST_IP:8500 in my case. This is why using consul-consul-server:8500 in the vault config WOULD work, but using HOST_IP:8500 would not. But using consul-consul-server:8500 causes other logging WARN messages to occur.

So fix HostPort or use a CNI that supports HostPort out of the box and this problem may go away for you.