search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.19k stars 4.42k forks source link

Consul works as a Vault backend, but with many false unhealthy members #6876

Open Aracki opened 4 years ago

Aracki commented 4 years ago

Even though Consul perfectly works as a Vault backend I am getting these errors in logs all the time:

devops-consul-devops-consul-server-2 consul     2019/12/04 08:36:44 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul     2019/12/04 08:36:45 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul     2019/12/04 08:36:46 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul     2019/12/04 08:36:47 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul     2019/12/04 08:36:48 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514

Consul info for both Client and Server

Server info ``` agent: check_monitors = 0 check_ttls = 4 checks = 4 services = 4 build: prerelease = revision = 1200f25e version = 1.6.2 consul: acl = disabled bootstrap = false known_datacenters = 1 leader = true leader_addr = 100.102.200.18:8300 server = true raft: applied_index = 129450973 commit_index = 129450973 fsm_pending = 0 last_contact = 0 last_log_index = 129450973 last_log_term = 45 last_snapshot_index = 129440101 last_snapshot_term = 45 latest_configuration = [{Suffrage:Voter ID:9c4f8adc-e40c-eed2-c291-bab53b9f94b4 Address:100.102.200.18:8300} {Suffrage:Voter ID:e6d7e185-c36f-cb45-cc1c-430e20b6cd71 Address:100.102.16.17:8300} {Suffrage:Voter ID:af37afcd-bdaa-8e52-fe2f-94b333e08b7f Address:100.104.176.16:8300}] latest_configuration_index = 128342841 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 45 runtime: arch = amd64 cpu_count = 16 goroutines = 243 max_procs = 16 os = linux version = go1.12.13 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 22 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 12 members = 3 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 12 members = 3 query_queue = 0 query_time = 1 ```

(For some reason on Consul UI, we have some duplicates and it randomly turns on and off) - but there are always at least 3 healthy checks according to the UI. image

Environment details

Consul version: 1.6.2 Vault version: 1.3.0

ndobbs commented 4 years ago

I too am experiencing the exact same issues as OP, same versions of consul and vault.

younesehb commented 4 years ago

I got the exact same error when deploying Vault and Consul with Helm.

Aracki commented 4 years ago

Has anyone tried this solution which includes deploying local consul agents? @ndobbs @younesehb