radykal-com
commented
1 year ago We are facing the same problem just with consul 1.15.1 (not using nomad)
consul troubleshoot proxy -upstream-ip=127.0.0.1
==> Validation
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
! Certificate chain is expired
-> Check the logs of the Consul agent configuring the local proxy and ensure XDS updates are being sent to the proxy
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 128536 connection failure(s)This wasn't happening with consul 1.14.4
kisunji
magnetarnix
mmeier86
jcpreston26
FelipeEmerim
jkirschner-hashicorp
Lord-Y
hashi-derek
david-yu
Overview of the Issue
I'm running a Nomad cluster using Consul Connect service mesh for almost all jobs. Since the update of Consul to 1.15.1 from 13.5, every three days, the Consul Connect mesh stops working. This seems to happen reliably 72 hours after a job (and its accompanying Envoy sidecar proxy) have been started.
I have also tried out the "consul troubleshoot proxy" command, and finally found a potential error source:
This seems to fit the "failing 72 hours after job start" symptoms, if indeed the problem is that the mTLS certs are not getting renewed.
Also note that I upgraded directly from 1.3.5 to 1.5.1, including the necessary changes for the breaking grpc_tls change. I did this relatively late update due to experiencing the Nomad config issues with gRPC and the Consul connect envoy proxy setup. Is it possible that I got the grpc_tls config wrong somewhere, and Consul is not able to update the Envoy proxy with new certificates?
One last note: Taking down the downstream job and restarting it (
nomad job stopandnomad job start) seems to completely fix the issue.Reproduction Steps
Steps to reproduce this issue, eg:
Consul info for both Client and Server
Client info
``` agent: check_monitors = 0 check_ttls = 0 checks = 11 services = 16 build: prerelease = revision = 7c04b6a0 version = 1.15.1 version_metadata = consul: acl = enabled known_servers = 3 server = false runtime: arch = amd64 cpu_count = 4 goroutines = 160 max_procs = 4 os = linux version = go1.20.1 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 147 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 67106 members = 16 query_queue = 0 query_time = 1 ``` ``` retry_join = ["server1", "server2", "server3"] server = false data_dir = "/path/to/data" addresses { https = "127.0.0.1" } advertise_addr = "10.***" datacenter = "homenet" log_file = "/var/log/consul/" log_rotate_max_files = 10 ports { http = -1 https = 8501 grpc_tls = 8502 } tls { defaults { cert_file = "/etc/homenet-certs/homenet.crt" key_file = "/etc/homenet-certs/homenet.priv" ca_file = "/etc/ssl/certs/homenet-ca.pem" } } acl = { enabled = true default_policy = "deny" enable_token_persistence = true tokens { agent = "agent-token" } } encrypt = "encryption-key" dns_config { node_ttl = "120s" service_ttl { "*" = "10s" } soa { min_ttl = 5 } } log_level = "info" connect { enabled = true } peering { enabled = false } ```Server info
``` agent: check_monitors = 0 check_ttls = 1 checks = 4 services = 4 build: prerelease = revision = 7c04b6a0 version = 1.15.1 version_metadata = consul: acl = enabled bootstrap = false known_datacenters = 1 leader = false leader_addr = 10.***:8300 server = true raft: applied_index = 40094594 commit_index = 40094594 fsm_pending = 0 last_contact = 58.855966ms last_log_index = 40094594 last_log_term = 356 last_snapshot_index = 40086560 last_snapshot_term = 356 latest_configuration = [{Suffrage:Voter ID:5d47ca91-6753-e044-4635-8fab8015d58c Address:server1:8300} {Suffrage:Voter ID:a412ed8e-3da1-a8c0-b968-b2d8e0327614 Address:server2:8300} {Suffrage:Voter ID:193eb5bd-3ddc-ca51-7b29 -9a1031a8b433 Address:server3:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 356 runtime: arch = arm64 cpu_count = 4 goroutines = 386 max_procs = 4 os = linux version = go1.20.1 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 147 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 67106 members = 16 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 2019 members = 3 query_queue = 0 query_time = 1 ``` ``` retry_join = ["server1", "server2", "server3"] server = true data_dir = "/data/dir" bootstrap_expect = 3 ui_config { enabled = true } addresses { dns = "0.0.0.0" https = "0.0.0.0" } advertise_addr = "10.***" datacenter = "homenet" log_file = "/var/log/consul/" log_rotate_max_files = 10 ports { http = -1 https = 8501 grpc_tls = 8502 } tls { defaults { cert_file = "/etc/homenet-certs/homenet.crt" key_file = "/etc/homenet-certs/homenet.priv" ca_file = "/etc/ssl/certs/homenet-ca.pem" } } acl = { enabled = true default_policy = "deny" enable_token_persistence = true tokens { agent = "****" # Special DNS token so that the Consul server can read all services and nodes # and answer DNS queries. default = "***" } } encrypt = "***" dns_config { node_ttl = "120s" service_ttl { "*" = "10s" } soa { min_ttl = 5 } } log_level = "info" connect { enabled = true } peering { enabled = false } ```Operating system and Environment details
Both the three servers and the clients are mostly running on Raspberry Pis, with a couple of x86 nodes in the mix as well.
Log Fragments
This is the big problem I'm seeing: There is nothing in any of the logs. Both the Consul servers and the Consul clients are only showing the standard "service check synced" messages. Just going by the logs, there is absolutely nothing wrong with the system. The same goes for the Envoy proxy sidecars in the Nomad jobs. They do not have any log output at all after initial startup.
Hm, looking further back in the logs, I saw this line in the Consul client log multiple times: