Closed MrAmbiG closed 2 years ago
is bug has any timeframe
I got the same issue, but after 2 to 3 minutes, the pods start properly 🤷♂️
Deployed to EKS as per instruction in https://www.vaultproject.io/docs/platform/k8s/helm/run but when it came to initializing the cluster I also got the same error.
I did the kubectl port-forward vault-0 8200:8200
but did not work...
$ kubectl exec -it vault-0 -- vault status Error checking seal status: Get http://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 1
I am also seeing this issue. Has anyone figured out what the issue is?
I am also running into this out of nowhere. Using the exact same Docker Image from vault:1.5.4 and exact same configuration (ie- values.yaml
) I never ran into this in other setups, but now I am tonight.
I got the same issue did any one has solution for that
I got the same issue . $ oc exec -it vault-0 vault status Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 1 when I look into pod logs I can see below 2020-11-04T12:23:17.945Z [WARN] storage migration check error: error="Get "http://10.160.225.18:8500/v1/kv/vault/core/migration": dial tcp 10.160.225.18:8500: connect: connection refused" So what I understood is , 10.160.225.18(HOST_IP) is my worked node where consul server pod is running , vault is not connecting to consul server with HOST_IP with 8500 port number , and below is my values .yaml storage "consul" { path = "vault/" address = "HOST_IP:8500"
what the work around I did was , I Changed HOST_IP:8500 to my consul SERVICE_IP , hence it is headless service , service ip not generated so I have given my consul service name , in my case my consul service name is "consul-server"
balaji@DESKTOP-O8C6N39:~/vault$ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-dns ClusterIP 172.21.33.6
values.yaml
storage "consul" { path = "vault" address = "consul-server:8500"
Then vault was deployed and working fine SO finally my vault HA with backend consul storage is working perfectly .
I have a similar problem when deploying Vault in HA mode with Consul as the storage backend configured to TLS as well with a self-signed CA. I simply generate the certs with the consul binary / CLI.
I was able to deploy Vault without TLS on top of the Consul backend, but I am continuously encountering the following connection refused issue when I deploy Vault with TLS enabled:
/ $ vault status
Error checking seal status: Get "https://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
/ $ vault operator init
Error initializing: Put "https://127.0.0.1:8200/v1/sys/init": dial tcp 127.0.0.1:8200: connect: connection refused
/ $
My consul helm values file looks as follows:
global:
enabled: true
domain: consul
name: consul
datacenter: consul-primary
tls:
# TLS must be enabled for federation in Kubernetes.
enabled: true
httpsOnly: true
enableAutoEncrypt: true
# Consul CA cert
caCert:
# The name of the Kubernetes secret.
secretName: consul-ca-cert
# The key of the Kubernetes secret.
secretKey: tls.crt
caKey:
# The name of the Kubernetes secret.
secretName: consul-ca-key
# The key of the Kubernetes secret.
secretKey: tls.key
server:
replicas: 3
bootstrapExpect: 3
# Consul server certs
extraVolumes:
- type: secret
name: consul-server-cert
load: true
ui:
enabled: true
service:
enabled: true
type: ClusterIP
My Vault values file looks as follows:
global:
enabled: true
tlsDisable: false
server:
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
extraVolumes:
- type: secret
name: vault-server-cert
- type: secret
name: consul-server-cert
- type: secret
name: consul-ca-cert
ha:
enabled: true
replicas: 3
config: |
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-cert/tls.crt"
tls_key_file = "/vault/userconfig/vault-server-cert/tls.key"
}
storage "consul" {
path = "vault"
address = "$HOST_IP:8501"
tls_key_file = "/vault/userconfig/consul-server-cert/tls.key"
tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt"
tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt"
}
service_registration "consul" {
address = "$HOST_IP:8501"
tls_key_file = "/vault/userconfig/consul-server-cert/tls.key"
tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt"
tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt"
}
ui:
enabled: true
serviceType: "ClusterIP"
externalPort: 8200
I have been stuck on this for days now given that none of the docs actually explains the process accurately... Please fix this.
Hi Vault is not able to communicating to backend consul host with that given port , so instead of HOST_IP:8501 , you can give your consul service ip or consul service name under the storage section in vault values.yaml file
storage "consul" { path = "vault" address = "your consul service ip or consul service name:8500" ( in my case I gave my consul service name ) tls_key_file = "/vault/userconfig/consul-server-cert/tls.key" tls_cert_file = "/vault/userconfig/consul-server-cert/tls.crt" tls_ca_file = "/vault/userconfig/consul-ca-cert/tls.crt" }
Hope this will resolve your issue ..
Just start consul along with it. The error will be resolved. ref:https://learn.hashicorp.com/tutorials/consul/kubernetes-deployment-guide?in=consul/kubernetes
worked using this solution! vault-values.yaml
server:
enabled: true
ha:
enabled: true
replicas: 3
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault/"
address = "consul-server:8500"
}
service_registration "kubernetes" {}
kubectl exec -it vault-0 -n vault -- vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.9.0
Storage Type consul
HA Enabled true
Thanks @muhammadmuhlas ! I used a namespace when installing the Consul helm chart, so the name of my Consul service was different than what the Vault helm chart defaults to. Your config showed me how to update that.
Vault HA using Helm on OpenShift. Getting this error.
Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
news ? I have the same issue
Hi,
Issues like this are often caused by Vault not being fully up and running (still initializing) or storage not being fully up and running. I'd double-check that each pod is healthy for your storage and Vault cluster, and check the logs on each to see if they are having some trouble initializing. If that doesn't work, I'd try again by following one of the tutorials.
If you have further issues, you might have better luck on the Vault discussion board.
Going to close for now as this doesn't appear to be a bug
I figured this one out for my cluster, its the CNI not supporting HostPort by default per: https://developer.hashicorp.com/consul/docs/troubleshoot/common-errors At the bottom of the page: "If you are still unable to connect and see i/o timeout or connection refused errors when connecting to the Consul client on the Kubernetes worker, this could be because the CNI (Container Networking Interface) does not support the use of hostPort."
I was using kube-router for CNI and it doesn't support HostPort by default. The error says connection is refused by 127.0.0.1:8200 but the issue is actually with connecting to HOST_IP:8500 in my case. This is why using consul-consul-server:8500 in the vault config WOULD work, but using HOST_IP:8500 would not. But using consul-consul-server:8500 causes other logging WARN messages to occur.
So fix HostPort or use a CNI that supports HostPort out of the box and this problem may go away for you.
kubectl exec -it vault-0 -- vault status Error checking seal status: Get http://localhost:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: connect: connection refused
.'kubectl exec -it vault-0 -- vault operator init -n 1 -t 1 Error initializing: Put http://127.0.0.1:8200/v1/sys/init: dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 2'
Official documentation i followed: https://www.hashicorp.com/blog/announcing-the-vault-helm-chart/