search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.44k stars 4.43k forks source link

Consul does not read a configuration file for the first 2 starts #21270

Closed EugenKon closed 5 months ago

EugenKon commented 5 months ago

Details are here: https://github.com/hashicorp/learn-nomad-cluster-setup/issues/39

EugenKon commented 5 months ago

So today I investigating more into this. I have terminated my instances and plan/apply.

Here I copied suspicious log lines:

Jun 07 14:22:38 ip-172-31-31-81 consul[935]:                Node ID: '36828085-505c-d4ea-ae89-21bcd145abbd'
Jun 07 14:21:10 ip-172-31-31-81 consul[371]:             Datacenter: 'dc1' (Segment: '')
Jun 07 14:21:10 ip-172-31-31-81 consul[371]:                 Server: false (Bootstrap: false)
Jun 07 14:21:10 ip-172-31-31-81 consul[371]:            ACL Enabled: false
Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.173Z [WARN]  agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.249Z [WARN]  agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
...
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: bootstrap = true: do not enable unless necessary
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ==> Starting Consul agent...
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]:                Node ID: '17e21bdf-fed3-33d4-f361-9575e1d0cc99'
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]:             Datacenter: 'dc1' (Segment: '<all>')
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]:                 Server: true (Bootstrap: true)
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]:            ACL Enabled: true
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.890Z [WARN]  agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.891Z [WARN]  agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.891Z [WARN]  agent: bootstrap = true: do not enable unless necessary

You can notice that on the first start there is no "earlier" reading of config file, but for the 3rd systemd consul start approach it is:

Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set

Also for the first two starts there are Server: false (Bootstrap: false) and ACL Enabled: false. In compare to the 3rd start it looks like the config was not read, because on the 3rd time we see Server: true (Bootstrap: true) and ACL Enabled: true.

I suppose that for the first two starts the configuration file was not read.

Consul log ``` Jun 07 14:21:04 ip-172-31-31-81 systemd[1]: Starting "HashiCorp Consul - A service mesh solution"... Jun 07 14:21:10 ip-172-31-31-81 consul[371]: ==> Starting Consul agent... Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Version: '1.18.2' Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Build Date: '2024-05-16 19:10:00 +0000 UTC' Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Node ID: '36828085-505c-d4ea-ae89-21bcd145abbd' Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Node name: 'ip-172-31-31-81' Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Datacenter: 'dc1' (Segment: '') Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Server: false (Bootstrap: false) Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600) Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Cluster Addr: 172.31.31.81 (LAN: 8301, WAN: 8302) Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Gossip Encryption: false Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Auto-Encrypt-TLS: false Jun 07 14:21:10 ip-172-31-31-81 consul[371]: ACL Enabled: false Jun 07 14:21:10 ip-172-31-31-81 consul[371]: ACL Default Policy: allow Jun 07 14:21:10 ip-172-31-31-81 consul[371]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2 Jun 07 14:21:10 ip-172-31-31-81 consul[371]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2 Jun 07 14:21:10 ip-172-31-31-81 consul[371]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2 Jun 07 14:21:10 ip-172-31-31-81 consul[371]: ==> Log data will now stream in as it occurs: Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.173Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.249Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.265Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: ip-172-31-31-81 172.31.31.81 Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.265Z [INFO] agent.router: Initializing LAN area manager Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.282Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.300Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.300Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.300Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/nodehealthstatus Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/workloadidentity Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/computedtrafficpermissions Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/computedexportedservices Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedimplicitdestinations Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedroutes Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/failoverpolicy Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/namespaceexportedservices Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/album Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/grpcroute Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/serviceendpoints Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/node Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/namespacetrafficpermissions Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/link Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshgateway Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/artist Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedproxyconfiguration Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedexplicitdestinations Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/service Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/healthstatus Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/partitionexportedservices Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/recordlabel Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/album Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/computedfailoverpolicy Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/trafficpermissions Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/tenancy/v2beta1/namespace Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/internal/v1/tombstone Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxystatetemplate Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.301Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.315Z [INFO] agent: started state syncer Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.315Z [INFO] agent: Consul agent running! Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.322Z [WARN] agent.router.manager: No servers available Jun 07 14:21:10 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:10.322Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.534Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.534Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.534Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.538Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.538Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.538Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.542Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.542Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.542Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.547Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.547Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.547Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.550Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.551Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.551Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.554Z [WARN] agent.router.manager: No servers available Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.554Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:11 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:11.554Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:15 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:15.047Z [ERROR] agent: Failed to check for updates: error="Get \"https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=0e9a0f73-8c65-788f-ca95-9fbe94d08749&version=1.18.2\": dial tcp: lookup checkpoint-api.hashicorp.com on 127.0.0.53:53: server misbehaving" Jun 07 14:21:19 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:19.211Z [WARN] agent: Check is now critical: check=_nomad-check-20a01bb9369a78e33355eefecf477b1cabab7f56 Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.402Z [WARN] agent.router.manager: No servers available Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.402Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.402Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.422Z [WARN] agent.router.manager: No servers available Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.422Z [WARN] agent: Syncing node info failed.: error="No known Consul servers" Jun 07 14:21:20 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:20.422Z [ERROR] agent: failed to sync changes: error="No known Consul servers" Jun 07 14:21:31 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:31.979Z [WARN] agent.router.manager: No servers available Jun 07 14:21:31 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:31.979Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:21:49 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:49.101Z [WARN] agent.router.manager: No servers available Jun 07 14:21:49 ip-172-31-31-81 consul[371]: 2024-06-07T14:21:49.101Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:22:18 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:18.522Z [WARN] agent.router.manager: No servers available Jun 07 14:22:18 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:18.522Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:22:34 ip-172-31-31-81 systemd[1]: consul.service: start operation timed out. Terminating. Jun 07 14:22:34 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:34.793Z [INFO] agent: Caught: signal=terminated Jun 07 14:22:34 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:34.793Z [INFO] agent: Gracefully shutting down agent... Jun 07 14:22:34 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:34.795Z [INFO] agent.client: client starting leave Jun 07 14:22:34 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:34.810Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: ip-172-31-31-81 172.31.31.81 Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.814Z [INFO] agent: Graceful exit completed Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.814Z [INFO] agent: Requesting shutdown Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.827Z [INFO] agent.client: shutting down client Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.843Z [INFO] agent: consul client down Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.843Z [INFO] agent: shutdown complete Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.843Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=tcp Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.843Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=udp Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.843Z [INFO] agent: Stopping server: address=127.0.0.1:8500 network=tcp protocol=http Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.848Z [INFO] agent: Waiting for endpoints to shut down Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.848Z [INFO] agent: Endpoints down Jun 07 14:22:37 ip-172-31-31-81 consul[371]: 2024-06-07T14:22:37.848Z [INFO] agent: Exit code: code=0 Jun 07 14:22:37 ip-172-31-31-81 systemd[1]: consul.service: Failed with result 'timeout'. Jun 07 14:22:37 ip-172-31-31-81 systemd[1]: Failed to start "HashiCorp Consul - A service mesh solution". Jun 07 14:22:38 ip-172-31-31-81 systemd[1]: consul.service: Scheduled restart job, restart counter is at 1. Jun 07 14:22:38 ip-172-31-31-81 systemd[1]: Stopped "HashiCorp Consul - A service mesh solution". Jun 07 14:22:38 ip-172-31-31-81 systemd[1]: Starting "HashiCorp Consul - A service mesh solution"... Jun 07 14:22:38 ip-172-31-31-81 consul[935]: ==> Starting Consul agent... Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Version: '1.18.2' Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Build Date: '2024-05-16 19:10:00 +0000 UTC' Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Node ID: '36828085-505c-d4ea-ae89-21bcd145abbd' Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Node name: 'ip-172-31-31-81' Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Datacenter: 'dc1' (Segment: '') Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Server: false (Bootstrap: false) Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600) Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Cluster Addr: 172.31.31.81 (LAN: 8301, WAN: 8302) Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Gossip Encryption: false Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Auto-Encrypt-TLS: false Jun 07 14:22:38 ip-172-31-31-81 consul[935]: ACL Enabled: false Jun 07 14:22:38 ip-172-31-31-81 consul[935]: ACL Default Policy: allow Jun 07 14:22:38 ip-172-31-31-81 consul[935]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2 Jun 07 14:22:38 ip-172-31-31-81 consul[935]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2 Jun 07 14:22:38 ip-172-31-31-81 consul[935]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2 Jun 07 14:22:38 ip-172-31-31-81 consul[935]: ==> Log data will now stream in as it occurs: Jun 07 14:22:38 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:38.959Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.006Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.024Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: ip-172-31-31-81 172.31.31.81 Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.024Z [INFO] agent.router: Initializing LAN area manager Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.044Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.044Z [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/recordlabel Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedproxyconfiguration Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshgateway Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/namespacetrafficpermissions Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.045Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/node Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxystatetemplate Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/grpcroute Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/tenancy/v2beta1/namespace Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/link Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.046Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/album Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.047Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.047Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/nodehealthstatus Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.047Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/workloadidentity Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.047Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/computedexportedservices Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.047Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/computedtrafficpermissions Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/internal/v1/tombstone Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedexplicitdestinations Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedroutes Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.050Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/namespaceexportedservices Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/artist Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/album Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/failoverpolicy Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedimplicitdestinations Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/service Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.051Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/serviceendpoints Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/healthstatus Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/computedfailoverpolicy Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/trafficpermissions Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/partitionexportedservices Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.052Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.056Z [INFO] agent: started state syncer Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.056Z [INFO] agent: Consul agent running! Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.059Z [WARN] agent.router.manager: No servers available Jun 07 14:22:39 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:39.059Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:22:47 ip-172-31-31-81 consul[935]: 2024-06-07T14:22:47.250Z [ERROR] agent: Failed to check for updates: error="Get \"https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=0e9a0f73-8c65-788f-ca95-9fbe94d08749&version=1.18.2\": dial tcp: lookup checkpoint-api.hashicorp.com on 127.0.0.53:53: server misbehaving" Jun 07 14:23:00 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:00.501Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:47330 error="ACL support disabled" Jun 07 14:23:05 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:05.142Z [WARN] agent.router.manager: No servers available Jun 07 14:23:05 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:05.142Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:23:05 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:05.594Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:47338 error="ACL support disabled" Jun 07 14:23:10 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:10.686Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:44150 error="ACL support disabled" Jun 07 14:23:15 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:15.768Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:44162 error="ACL support disabled" Jun 07 14:23:16 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:16.333Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: ip-172-31-4-167 172.31.4.167 Jun 07 14:23:20 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:20.728Z [WARN] agent.router.manager: No servers available Jun 07 14:23:20 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:20.728Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:23:20 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:20.851Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:59942 error="ACL support disabled" Jun 07 14:23:25 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:25.932Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:59958 error="ACL support disabled" Jun 07 14:23:31 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:31.012Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:41814 error="ACL support disabled" Jun 07 14:23:36 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:36.088Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:41828 error="ACL support disabled" Jun 07 14:23:41 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:41.173Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:39116 error="ACL support disabled" Jun 07 14:23:46 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:46.266Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:39128 error="ACL support disabled" Jun 07 14:23:48 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:48.512Z [WARN] agent.router.manager: No servers available Jun 07 14:23:48 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:48.512Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:23:51 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:51.353Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:45736 error="ACL support disabled" Jun 07 14:23:56 ip-172-31-31-81 consul[935]: 2024-06-07T14:23:56.434Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:45748 error="ACL support disabled" Jun 07 14:24:01 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:01.515Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:49330 error="ACL support disabled" Jun 07 14:24:06 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:06.591Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:49346 error="ACL support disabled" Jun 07 14:24:08 ip-172-31-31-81 systemd[1]: consul.service: start operation timed out. Terminating. Jun 07 14:24:08 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:08.292Z [INFO] agent: Caught: signal=terminated Jun 07 14:24:08 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:08.292Z [INFO] agent: Gracefully shutting down agent... Jun 07 14:24:08 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:08.292Z [INFO] agent.client: client starting leave Jun 07 14:24:09 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:09.025Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: ip-172-31-31-81 172.31.31.81 Jun 07 14:24:11 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:11.669Z [ERROR] agent.http: Request error: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:39442 error="ACL support disabled" Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.110Z [WARN] agent.router.manager: No servers available Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.110Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers" Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.626Z [INFO] agent: Graceful exit completed Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.626Z [INFO] agent: Requesting shutdown Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.627Z [INFO] agent.client: shutting down client Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.630Z [INFO] agent: consul client down Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.630Z [INFO] agent: shutdown complete Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.630Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.631Z [INFO] agent.dns: Stopping server: protocol=DNS address=127.0.0.1:8600 network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.631Z [INFO] agent: Stopping server: address=127.0.0.1:8500 network=tcp protocol=http Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.631Z [INFO] agent: Waiting for endpoints to shut down Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.631Z [INFO] agent: Endpoints down Jun 07 14:24:12 ip-172-31-31-81 consul[935]: 2024-06-07T14:24:12.631Z [INFO] agent: Exit code: code=0 Jun 07 14:24:12 ip-172-31-31-81 systemd[1]: consul.service: Failed with result 'timeout'. Jun 07 14:24:12 ip-172-31-31-81 systemd[1]: Failed to start "HashiCorp Consul - A service mesh solution". Jun 07 14:24:12 ip-172-31-31-81 systemd[1]: consul.service: Scheduled restart job, restart counter is at 2. Jun 07 14:24:12 ip-172-31-31-81 systemd[1]: Stopped "HashiCorp Consul - A service mesh solution". ```
Consul configuration ``` datacenter = "dc1" data_dir = "/opt/consul/data" server = true bootstrap_expect = 1 retry_join = ["provider=aws tag_key=ConsulAutoJoin tag_value=nomad-auto-join"] domain = "xxx.internal" recursors = [ "169.254.169.253" ] client_addr = "0.0.0.0" bind_addr = "0.0.0.0" advertise_addr = "172.31.31.81" acl { enabled = true default_policy = "deny" down_policy = "extend-cache" } enable_debug = true log_level = "trace" enable_syslog = true log_rotate_max_files = 30 log_rotate_duration = "24h" ui_config { enabled = true } tls { defaults { tls_min_version = "TLSv1_3" } } ports { dns = 8600 grpc = 8502 } connect { enabled = true } services { id = "dns" name = "dns" tags = ["primary"] address = "localhost" port = 8600 check { id = "dns" name = "Consul DNS TCP on port 8600" tcp = "localhost:8600" interval = "10s" timeout = "1s" } } services { name = "consul" } ```
EugenKon commented 5 months ago
Consul log for 3rd try ``` Jun 07 14:24:12 ip-172-31-31-81 systemd[1]: Starting "HashiCorp Consul - A service mesh solution"... Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode. Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: bootstrap = true: do not enable unless necessary Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: Update: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: OutgoingRPCWrapper: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: OutgoingALPNRPCWrapper: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] original dial target is: "consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] parsed dial target is: {Scheme:consul Authority:dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99 URL:{Scheme:consul Opaque: User: Host:dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99 Path:/server.dc1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel authority set to "server.dc1" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Resolver state updated: { "Addresses": null, "ServiceConfig": null, "Attributes": null } () Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel switches to new LB policy "consul-internal" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: creating balancer: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: switching server: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 from= to= Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel Connectivity change to TRANSIENT_FAILURE Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ==> Starting Consul agent... Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Version: '1.18.2' Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Build Date: '2024-05-16 19:10:00 +0000 UTC' Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Node ID: '17e21bdf-fed3-33d4-f361-9575e1d0cc99' Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Node name: 'ip-172-31-31-81' Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Datacenter: 'dc1' (Segment: '') Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Server: true (Bootstrap: true) Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600) Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Cluster Addr: 172.31.31.81 (LAN: 8301, WAN: 8302) Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Gossip Encryption: false Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Auto-Encrypt-TLS: false Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ACL Enabled: true Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Reporting Enabled: false Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ACL Default Policy: deny Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_3 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_3 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_3 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ==> Log data will now stream in as it occurs: Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.890Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.891Z [WARN] agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode. Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.891Z [WARN] agent: bootstrap = true: do not enable unless necessary Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.892Z [TRACE] agent.tlsutil: Update: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.892Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.892Z [TRACE] agent.tlsutil: OutgoingALPNRPCWrapper: version=1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.892Z [TRACE] agent: [core][Channel #1] Channel created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.893Z [TRACE] agent: [core][Channel #1] original dial target is: "consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.893Z [TRACE] agent: [core][Channel #1] parsed dial target is: {Scheme:consul Authority:dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99 URL:{Scheme:consul Opaque: User: Host:dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99 Path:/server.dc1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.893Z [TRACE] agent: [core][Channel #1] Channel authority set to "server.dc1" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.893Z [TRACE] agent: [core][Channel #1] Resolver state updated: { Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Addresses": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "ServiceConfig": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Attributes": null Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: } () Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.893Z [TRACE] agent: [core][Channel #1] Channel switches to new LB policy "consul-internal" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.894Z [TRACE] agent.grpc.balancer: creating balancer: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.894Z [DEBUG] agent.grpc.balancer: switching server: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 from= to= Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.894Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to TRANSIENT_FAILURE Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.904Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.904Z [WARN] agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode. Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode. Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.auto_config: bootstrap = true: do not enable unless necessary Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.905Z [TRACE] agent.tlsutil: Update: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: Update: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.906Z [TRACE] agent.tlsutil: IncomingGRPConfig: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: IncomingGRPConfig: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.907Z [TRACE] agent: [core][Server #2] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #2] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.915Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.tlsutil: OutgoingRPCWrapper: version=2 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.927Z [TRACE] agent: [core][Server #3] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #3] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.929Z [TRACE] agent: [core][Server #3 ListenSocket #4] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #3 ListenSocket #4] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.930Z [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:17e21bdf-fed3-33d4-f361-9575e1d0cc99 Address:172.31.31.81:8300}]" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:17e21bdf-fed3-33d4-f361-9575e1d0cc99 Address:172.31.31.81:8300}]" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.931Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: ip-172-31-31-81.dc1 172.31.31.81 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.serf.wan: serf: EventMemberJoin: ip-172-31-31-81.dc1 172.31.31.81 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.932Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: ip-172-31-31-81 172.31.31.81 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: EventMemberJoin: ip-172-31-31-81 172.31.31.81 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.932Z [INFO] agent.router: Initializing LAN area manager Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.router: Initializing LAN area manager Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.933Z [TRACE] agent: [core][Channel #1] Resolver state updated: { Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Addresses": [ Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: { Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Addr": "dc1-172.31.31.81:8300", Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "ServerName": "ip-172-31-31-81", Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Attributes": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "BalancerAttributes": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Type": 0, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Metadata": null Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: } Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ], Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "ServiceConfig": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Attributes": null Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: } (resolver returned new addresses) Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Resolver state updated: { "Addresses": [ { "Addr": "dc1-172.31.31.81:8300", "ServerName": "ip-172-31-31-81", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null } ], "ServiceConfig": null, "Attributes": null } (resolver returned new addresses) Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.934Z [TRACE] agent.grpc.balancer: adding server address: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 address=dc1-172.31.31.81:8300 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: adding server address: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 address=dc1-172.31.31.81:8300 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.934Z [DEBUG] agent.grpc.balancer: switching server: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 from= to=dc1-172.31.31.81:8300 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: switching server: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 from= to=dc1-172.31.31.81:8300 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.934Z [TRACE] agent: [core][Channel #1 SubChannel #5] Subchannel created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1 SubChannel #5] Subchannel created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.935Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel Connectivity change to CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.935Z [TRACE] agent: [core][Server #6] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #6] Server created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.936Z [INFO] agent.server.autopilot: reconciliation now disabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.autopilot: reconciliation now disabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.936Z [INFO] agent.server.raft: entering follower state: follower="Node at 172.31.31.81:8300 [Follower]" leader-address= leader-id= Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.raft: entering follower state: follower="Node at 172.31.31.81:8300 [Follower]" leader-address= leader-id= Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.936Z [TRACE] agent: [core][Channel #1 SubChannel #5] Subchannel Connectivity change to CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1 SubChannel #5] Subchannel Connectivity change to CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.940Z [TRACE] agent: [core][Channel #1 SubChannel #5] Subchannel picks a new address "dc1-172.31.31.81:8300" to connect Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1 SubChannel #5] Subchannel picks a new address "dc1-172.31.31.81:8300" to connect Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.942Z [TRACE] agent: [core][Server #6 ListenSocket #7] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #6 ListenSocket #7] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.936Z [INFO] agent.server: Adding LAN server: server="ip-172-31-31-81 (Addr: tcp/172.31.31.81:8300) (DC: dc1)" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server: Adding LAN server: server="ip-172-31-31-81 (Addr: tcp/172.31.31.81:8300) (DC: dc1)" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.944Z [TRACE] agent: [core][Channel #1] Resolver state updated: { Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Addresses": [ Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: { Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Addr": "dc1-172.31.31.81:8300", Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "ServerName": "ip-172-31-31-81", Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Attributes": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "BalancerAttributes": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Type": 0, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Metadata": null Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: } Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: ], Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "ServiceConfig": null, Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: "Attributes": null Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: } () Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Resolver state updated: { "Addresses": [ { "Addr": "dc1-172.31.31.81:8300", "ServerName": "ip-172-31-31-81", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null } ], "ServiceConfig": null, "Attributes": null } () Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.945Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 server=dc1-172.31.31.81:8300 state=CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: sub-connection state changed: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 server=dc1-172.31.31.81:8300 state=CONNECTING Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.936Z [INFO] agent.server: Handled event for server in area: event=member-join server=ip-172-31-31-81.dc1 area=wan Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server: Handled event for server in area: event=member-join server=ip-172-31-31-81.dc1 area=wan Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.949Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.951Z [INFO] agent.server.cert-manager: initialized server certificate management Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: initialized server certificate management Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.951Z [DEBUG] agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.952Z [DEBUG] agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.952Z [DEBUG] agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: recursor enabled Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.952Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Started DNS server: address=0.0.0.0:8600 network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.953Z [DEBUG] agent.server.autopilot: autopilot is now running Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.autopilot: autopilot is now running Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.953Z [DEBUG] agent.server.autopilot: state update routine is now running Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.server.autopilot: state update routine is now running Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.953Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.proxycfg.agent-state: syncing proxy services from local state Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.954Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Started DNS server: address=0.0.0.0:8600 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.954Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/trafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/trafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.955Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.955Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedexplicitdestinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedexplicitdestinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.955Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.955Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/namespacetrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/namespacetrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/partitionexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/partitionexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedproxyconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/node Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/workloadidentity Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/tenancy/v2beta1/namespace Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/link Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/recordlabel Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/grpcroute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshgateway Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/computedfailoverpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/computedexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedroutes Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedproxyconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedimplicitdestinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/serviceendpoints Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/failoverpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/computedtrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.956Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.957Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/internal/v1/tombstone Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/node Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/workloadidentity Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/tenancy/v2beta1/namespace Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/link Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v1/recordlabel Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/grpcroute Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshgateway Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/computedfailoverpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/computedexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedroutes Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/computedimplicitdestinations Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/serviceendpoints Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/failoverpolicy Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/computedtrafficpermissions Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/internal/v1/tombstone Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/album Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/service Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/healthstatus Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/nodehealthstatus Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/artist Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxystatetemplate Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/namespaceexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/album Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: started state syncer Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: Consul agent running! Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: Started gRPC listeners: port_name=grpc address=[::]:8502 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [TRACE] agent: [core][Server #2 ListenSocket #10] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8503 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [TRACE] agent: [core][Server #2 ListenSocket #11] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.960Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.961Z [INFO] agent: Joining cluster...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v1/album Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.961Z [DEBUG] agent: discover: Using provider "aws": cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.961Z [INFO] agent: discover-aws: Service type is not supported. Valid values are {ec2,ecs}. Falling back to 'ec2': cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.962Z [INFO] agent: discover-aws: Address type is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.962Z [DEBUG] agent: discover-aws: Using region= tag_key=ConsulAutoJoin tag_value=nomad-auto-join addr_type=private_v4: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.962Z [DEBUG] agent: discover-aws: No static credentials: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.962Z [DEBUG] agent: discover-aws: Using environment variables, shared credentials or instance role: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.962Z [INFO] agent: discover-aws: Region not provided. Looking up region in ec2 metadata...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/service Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/healthstatus Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/nodehealthstatus Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v1/artist Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxystatetemplate Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/namespaceexportedservices Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.http: Registered resource endpoint: endpoint=/api/demo/v2/album Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Starting server: address=[::]:8500 network=tcp protocol=http Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: started state syncer Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Consul agent running! Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Started gRPC listeners: port_name=grpc address=[::]:8502 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.965Z [TRACE] agent: [core][Channel #1 SubChannel #5] Subchannel Connectivity change to READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.965Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 server=dc1-172.31.31.81:8300 state=READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.965Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #2 ListenSocket #10] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8503 network=tcp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Server #2 ListenSocket #11] ListenSocket created Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: Joining cluster...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover: Using provider "aws": cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Service type is not supported. Valid values are {ec2,ecs}. Falling back to 'ec2': cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Address type is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Using region= tag_key=ConsulAutoJoin tag_value=nomad-auto-join addr_type=private_v4: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: No static credentials: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Using environment variables, shared credentials or instance role: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Region not provided. Looking up region in ec2 metadata...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1 SubChannel #5] Subchannel Connectivity change to READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.grpc.balancer: sub-connection state changed: target=consul://dc1.17e21bdf-fed3-33d4-f361-9575e1d0cc99/server.dc1 server=dc1-172.31.31.81:8300 state=READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: [core][Channel #1] Channel Connectivity change to READY Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Region is us-west-2: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.968Z [INFO] agent: discover-aws: Region is us-west-2: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.968Z [DEBUG] agent: discover-aws: Creating session...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.968Z [INFO] agent: discover-aws: Filter instances with ConsulAutoJoin=nomad-auto-join: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Creating session...: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent: discover-aws: Filter instances with ConsulAutoJoin=nomad-auto-join: cluster=LAN Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.979Z [DEBUG] agent.dns: recurse succeeded for question: question="{ec2.us-west-2.amazonaws.com. 1 1}" rtt=2.452318ms recursor=169.254.169.253:53 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{ec2.us-west-2.amazonaws.com. 1 1}" rtt=2.452318ms recursor=169.254.169.253:53 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.980Z [DEBUG] agent.dns: request served from client: question="{ec2.us-west-2.amazonaws.com. 1 1}" network=udp latency=3.156123ms client=127.0.0.1:40118 client_network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{ec2.us-west-2.amazonaws.com. 1 1}" network=udp latency=3.156123ms client=127.0.0.1:40118 client_network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.980Z [DEBUG] agent.dns: recurse succeeded for question: question="{ec2.us-west-2.amazonaws.com. 28 1}" rtt=3.398322ms recursor=169.254.169.253:53 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{ec2.us-west-2.amazonaws.com. 28 1}" rtt=3.398322ms recursor=169.254.169.253:53 Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:12.981Z [DEBUG] agent.dns: request served from client: question="{ec2.us-west-2.amazonaws.com. 28 1}" network=udp latency=4.031888ms client=127.0.0.1:53758 client_network=udp Jun 07 14:24:12 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{ec2.us-west-2.amazonaws.com. 28 1}" network=udp latency=4.031888ms client=127.0.0.1:53758 client_network=udp Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.229Z [DEBUG] agent: discover-aws: Found 2 reservations: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Found 2 reservations: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.230Z [DEBUG] agent: discover-aws: Reservation r-0d773400043717daf has 1 instances: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Reservation r-0d773400043717daf has 1 instances: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.230Z [DEBUG] agent: discover-aws: Found instance i-053b577e8e9387b1d: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Found instance i-053b577e8e9387b1d: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.231Z [INFO] agent: discover-aws: Instance i-053b577e8e9387b1d has private ip 172.31.4.167: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Instance i-053b577e8e9387b1d has private ip 172.31.4.167: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.231Z [DEBUG] agent: discover-aws: Reservation r-0daa0b4c16d6df44c has 1 instances: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Reservation r-0daa0b4c16d6df44c has 1 instances: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.231Z [DEBUG] agent: discover-aws: Found instance i-04d4aacaff4462431: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Found instance i-04d4aacaff4462431: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.232Z [INFO] agent: discover-aws: Instance i-04d4aacaff4462431 has private ip 172.31.31.81: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Instance i-04d4aacaff4462431 has private ip 172.31.31.81: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.232Z [DEBUG] agent: discover-aws: Found ip addresses: [172.31.4.167 172.31.31.81]: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: discover-aws: Found ip addresses: [172.31.4.167 172.31.31.81]: cluster=LAN Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.232Z [INFO] agent: Discovered servers: cluster=LAN cluster=LAN servers="172.31.4.167 172.31.31.81" Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: Discovered servers: cluster=LAN cluster=LAN servers="172.31.4.167 172.31.31.81" Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.232Z [INFO] agent: (LAN) joining: lan_addresses=["172.31.4.167", "172.31.31.81"] Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: (LAN) joining: lan_addresses=["172.31.4.167", "172.31.31.81"] Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.233Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 172.31.4.167:8301 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 172.31.4.167:8301 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.235Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: ip-172-31-4-167 172.31.4.167 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: EventMemberJoin: ip-172-31-4-167 172.31.4.167 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.236Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a dead message (from: ip-172-31-31-81) Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.memberlist.lan: memberlist: Refuting a dead message (from: ip-172-31-31-81) Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.236Z [DEBUG] agent.server.serf.lan: serf: Refuting an older leave intent Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: Refuting an older leave intent Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.236Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 172.31.31.81:8301 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 172.31.31.81:8301 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.238Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=172.31.31.81:49302 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.memberlist.lan: memberlist: Stream connection from=172.31.31.81:49302 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.240Z [INFO] agent: (LAN) joined: number_of_nodes=2 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: (LAN) joined: number_of_nodes=2 Jun 07 14:24:13 ip-172-31-31-81 systemd[1]: Started "HashiCorp Consul - A service mesh solution". Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.241Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.530Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.530Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.730Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.731Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.930Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.931Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.933Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.934Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageJoinType: ip-172-31-31-81 Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.954Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:13.954Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:13 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:14.955Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:14.955Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:14.956Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:14 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:15 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:15.956Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:15 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:15 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:15.957Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:15 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:16.955Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:16.957Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:16.958Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:16 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:17.322Z [DEBUG] agent: Check status updated: check=dns status=passing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: agent: Check status updated: check=dns status=passing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:17.959Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:17.960Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:17 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:18.918Z [TRACE] agent.server: rpc_server_call: method=Catalog.ListDatacenters errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Catalog.ListDatacenters errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:18.955Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:18.960Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:18.961Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:18 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:19 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:19.962Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:19 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:19 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:19.963Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:19 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.448Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.449Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency=1.057651ms Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency=1.057651ms Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.450Z [DEBUG] agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.450Z [DEBUG] agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.451Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency=1.691316ms Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency=1.691316ms Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.452Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.453Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="866.32µs" Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="866.32µs" Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.954Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.964Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:20.964Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:20 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.457Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.457Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency=1.745765ms Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency=1.745765ms Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.458Z [DEBUG] agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.458Z [DEBUG] agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.459Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency="777.397µs" Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency="777.397µs" Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.459Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.460Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="581.745µs" Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="581.745µs" Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.936Z [TRACE] agent.server.usage_metrics: Starting usage run Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.server.usage_metrics: Starting usage run Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.965Z [DEBUG] agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: ACLs have not finished initializing Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:21.966Z [DEBUG] agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:21 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA has not finished initializing Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.232Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.232Z [INFO] agent.server.raft: entering candidate state: node="Node at 172.31.31.81:8300 [Candidate]" term=2 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: entering candidate state: node="Node at 172.31.31.81:8300 [Candidate]" term=2 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.235Z [DEBUG] agent.server.raft: voting for self: term=2 id=17e21bdf-fed3-33d4-f361-9575e1d0cc99 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: voting for self: term=2 id=17e21bdf-fed3-33d4-f361-9575e1d0cc99 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.238Z [DEBUG] agent.server.raft: calculated votes needed: needed=1 term=2 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: calculated votes needed: needed=1 term=2 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.238Z [DEBUG] agent.server.raft: vote granted: from=17e21bdf-fed3-33d4-f361-9575e1d0cc99 term=2 tally=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: vote granted: from=17e21bdf-fed3-33d4-f361-9575e1d0cc99 term=2 tally=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.238Z [INFO] agent.server.raft: election won: term=2 tally=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: election won: term=2 tally=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.239Z [INFO] agent.server.raft: entering leader state: leader="Node at 172.31.31.81:8300 [Leader]" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.raft: entering leader state: leader="Node at 172.31.31.81:8300 [Leader]" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.241Z [DEBUG] agent.server.controller-runtime: controller running: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: controller running: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.241Z [DEBUG] agent.server.controller-runtime: priming caches: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: priming caches: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.242Z [DEBUG] agent.server.controller-runtime: priming caches complete: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: priming caches complete: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.243Z [INFO] agent.server: cluster leadership acquired Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: cluster leadership acquired Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.245Z [INFO] agent.server: initializing acls Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: initializing acls Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.249Z [INFO] agent.server: Created ACL 'global-management' policy Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: Created ACL 'global-management' policy Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.254Z [INFO] agent.server: Created ACL 'builtin/global-read-only' policy Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: Created ACL 'builtin/global-read-only' policy Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.257Z [INFO] agent.server: Created ACL anonymous token from configuration Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: Created ACL anonymous token from configuration Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.259Z [TRACE] agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.261Z [INFO] agent.leader: started routine: routine="acl token reaping" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="acl token reaping" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.262Z [DEBUG] agent.server.controller-runtime: controller running: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: controller running: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.264Z [DEBUG] agent.server.controller-runtime: priming caches: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: priming caches: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.265Z [INFO] agent.server: New leader elected: payload=ip-172-31-31-81 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: New leader elected: payload=ip-172-31-31-81 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.265Z [TRACE] agent.server: rpc_server_call: method=AutopilotRequest.Apply errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=AutopilotRequest.Apply errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.266Z [INFO] agent.server.autopilot: reconciliation now enabled Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.autopilot: reconciliation now enabled Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.267Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="federation state anti-entropy" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.267Z [INFO] agent.leader: started routine: routine="federation state pruning" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="federation state pruning" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.267Z [INFO] agent.leader: started routine: routine="streaming peering resources" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="streaming peering resources" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.268Z [INFO] agent.leader: started routine: routine="metrics for streaming peering resources" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="metrics for streaming peering resources" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.268Z [INFO] agent.leader: started routine: routine="peering deferred deletion" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="peering deferred deletion" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.266Z [DEBUG] agent.server.controller-runtime: priming caches complete: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.controller-runtime: priming caches complete: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.269Z [TRACE] agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.270Z [TRACE] agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.270Z [TRACE] agent.server.peering-syncer: blocking for changes: sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: blocking for changes: sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.275Z [DEBUG] connect.ca.consul: consul CA provider configured: id=fb:50:9b:45:1a:65:15:c1:68:57:73:5f:da:cd:b8:0d:0f:e2:26:eb:68:66:43:11:85:9d:67:a9:7a:56:9c:b9 is_primary=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: connect.ca.consul: consul CA provider configured: id=fb:50:9b:45:1a:65:15:c1:68:57:73:5f:da:cd:b8:0d:0f:e2:26:eb:68:66:43:11:85:9d:67:a9:7a:56:9c:b9 is_primary=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.275Z [TRACE] agent.server.peering-syncer: unblocked: sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: unblocked: sequence_id=befd648b-de7c-cfb2-d9e8-5b6a8710124b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.276Z [TRACE] agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.276Z [TRACE] agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.276Z [TRACE] agent.server.peering-syncer: blocking for changes: sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: blocking for changes: sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.279Z [TRACE] agent.server.peering-syncer: unblocked: sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: unblocked: sequence_id=1f5c28cd-332b-98ca-7cfb-104aa54d4cb8 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.279Z [TRACE] agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.280Z [TRACE] agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.280Z [TRACE] agent.server.peering-syncer: blocking for changes: sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: blocking for changes: sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.283Z [TRACE] agent.server.peering-syncer: unblocked: sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: unblocked: sequence_id=66059c62-3634-c347-7809-1675256fa57b Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.284Z [TRACE] agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.284Z [TRACE] agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.284Z [TRACE] agent.server.peering-syncer: blocking for changes: sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: blocking for changes: sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.287Z [INFO] connect.ca: updated root certificates from primary datacenter Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: connect.ca: updated root certificates from primary datacenter Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.288Z [INFO] connect.ca: initialized primary datacenter CA with provider: provider=consul Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: connect.ca: initialized primary datacenter CA with provider: provider=consul Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.288Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="intermediate cert renew watch" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.288Z [INFO] agent.leader: started routine: routine="CA root pruning" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="CA root pruning" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.289Z [INFO] agent.leader: started routine: routine="CA root expiration metric" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="CA root expiration metric" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.289Z [INFO] agent.leader: started routine: routine="CA signing expiration metric" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="CA signing expiration metric" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.289Z [INFO] agent.leader: started routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.290Z [INFO] agent.leader: started routine: routine="config entry controllers" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: started routine: routine="config entry controllers" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.290Z [TRACE] agent.server.peering-syncer: unblocked: sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: unblocked: sequence_id=410bc899-5237-47ae-2111-874108349418 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.291Z [TRACE] agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: syncing new list of peers: num_peers=0 sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.291Z [TRACE] agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: checking connected streams: streams=map[] sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.292Z [TRACE] agent.server.peering-syncer: blocking for changes: sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: blocking for changes: sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.292Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=APIGateway Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=APIGateway Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.293Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=HTTPRoute Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=HTTPRoute Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.293Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=TCPRoute Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=TCPRoute Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.293Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=BoundAPIGateway Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=BoundAPIGateway Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.294Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=InlineCertificate Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=InlineCertificate Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.294Z [DEBUG] agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=JWTProvider Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.api_gateway_controller: rewatching controller subscription: controller=apiGatewayController topic=JWTProvider Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.297Z [TRACE] agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.298Z [TRACE] agent.server.peering-syncer: unblocked: sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.peering-syncer: unblocked: sequence_id=d0cb8b5e-56fe-4985-53a3-8738d6acd7d5 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.299Z [TRACE] agent.server: rpc_server_call: method=Intentions.DeleteAll errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Intentions.DeleteAll errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.299Z [DEBUG] agent.server: successfully established leadership: duration=54.230675ms Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: successfully established leadership: duration=54.230675ms Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.299Z [INFO] agent.server: member joined, marking health alive: member=ip-172-31-31-81 partition=default Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: member joined, marking health alive: member=ip-172-31-31-81 partition=default Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.301Z [TRACE] agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=SystemMetadata.Upsert errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.302Z [INFO] agent.leader: stopping routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: stopping routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.302Z [INFO] agent.leader: stopped routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.leader: stopped routine: routine="virtual IP version check" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.307Z [INFO] agent.server: member joined, marking health alive: member=ip-172-31-4-167 partition=default Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: member joined, marking health alive: member=ip-172-31-4-167 partition=default Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.308Z [DEBUG] agent.server.xds_capacity_controller: updating drain rate limit: rate_limit=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.xds_capacity_controller: updating drain rate limit: rate_limit=1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.452Z [DEBUG] agent.dns: recurse succeeded for question: question="{ntp.ubuntu.com. 28 1}" rtt=1.806884ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{ntp.ubuntu.com. 28 1}" rtt=1.806884ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.453Z [DEBUG] agent.dns: request served from client: question="{ntp.ubuntu.com. 28 1}" network=udp latency=2.578785ms client=127.0.0.1:60891 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{ntp.ubuntu.com. 28 1}" network=udp latency=2.578785ms client=127.0.0.1:60891 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.453Z [DEBUG] agent.dns: recurse succeeded for question: question="{ntp.ubuntu.com. 1 1}" rtt=2.822149ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{ntp.ubuntu.com. 1 1}" rtt=2.822149ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.454Z [DEBUG] agent.dns: request served from client: question="{ntp.ubuntu.com. 1 1}" network=udp latency=3.476416ms client=127.0.0.1:47546 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{ntp.ubuntu.com. 1 1}" network=udp latency=3.476416ms client=127.0.0.1:47546 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.457Z [DEBUG] agent.dns: recurse succeeded for question: question="{0.ubuntu.pool.ntp.org. 28 1}" rtt=1.758103ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{0.ubuntu.pool.ntp.org. 28 1}" rtt=1.758103ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.458Z [DEBUG] agent.dns: request served from client: question="{0.ubuntu.pool.ntp.org. 28 1}" network=udp latency=2.401058ms client=127.0.0.1:42855 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{0.ubuntu.pool.ntp.org. 28 1}" network=udp latency=2.401058ms client=127.0.0.1:42855 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.458Z [DEBUG] agent.dns: recurse succeeded for question: question="{0.ubuntu.pool.ntp.org. 1 1}" rtt=2.649914ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{0.ubuntu.pool.ntp.org. 1 1}" rtt=2.649914ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.459Z [DEBUG] agent.dns: request served from client: question="{0.ubuntu.pool.ntp.org. 1 1}" network=udp latency=3.752211ms client=127.0.0.1:46112 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{0.ubuntu.pool.ntp.org. 1 1}" network=udp latency=3.752211ms client=127.0.0.1:46112 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.462Z [DEBUG] agent.dns: recurse succeeded for question: question="{1.ubuntu.pool.ntp.org. 1 1}" rtt=1.96331ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{1.ubuntu.pool.ntp.org. 1 1}" rtt=1.96331ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.462Z [DEBUG] agent.dns: request served from client: question="{1.ubuntu.pool.ntp.org. 1 1}" network=udp latency=2.594971ms client=127.0.0.1:34812 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{1.ubuntu.pool.ntp.org. 1 1}" network=udp latency=2.594971ms client=127.0.0.1:34812 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.463Z [DEBUG] agent.dns: recurse succeeded for question: question="{1.ubuntu.pool.ntp.org. 28 1}" rtt=3.067342ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{1.ubuntu.pool.ntp.org. 28 1}" rtt=3.067342ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.464Z [DEBUG] agent.dns: request served from client: question="{1.ubuntu.pool.ntp.org. 28 1}" network=udp latency=4.246121ms client=127.0.0.1:59848 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{1.ubuntu.pool.ntp.org. 28 1}" network=udp latency=4.246121ms client=127.0.0.1:59848 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.466Z [DEBUG] agent.dns: recurse succeeded for question: question="{2.ubuntu.pool.ntp.org. 28 1}" rtt=1.631197ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{2.ubuntu.pool.ntp.org. 28 1}" rtt=1.631197ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.466Z [DEBUG] agent.dns: request served from client: question="{2.ubuntu.pool.ntp.org. 28 1}" network=udp latency=2.224081ms client=127.0.0.1:37730 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{2.ubuntu.pool.ntp.org. 28 1}" network=udp latency=2.224081ms client=127.0.0.1:37730 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.467Z [DEBUG] agent.dns: recurse succeeded for question: question="{2.ubuntu.pool.ntp.org. 1 1}" rtt=2.441261ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: recurse succeeded for question: question="{2.ubuntu.pool.ntp.org. 1 1}" rtt=2.441261ms recursor=169.254.169.253:53 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.468Z [DEBUG] agent.dns: request served from client: question="{2.ubuntu.pool.ntp.org. 1 1}" network=udp latency=3.266899ms client=127.0.0.1:35096 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.dns: request served from client: question="{2.ubuntu.pool.ntp.org. 1 1}" network=udp latency=3.266899ms client=127.0.0.1:35096 client_network=udp Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.530Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.693Z [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=true request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Coordinate.Update errored=true request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.713Z [TRACE] agent.server: rpc_server_call: method=FederationState.Apply errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=FederationState.Apply errored=false request_type=unreported rpc_type=internal leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.714Z [INFO] agent.server: federation state anti-entropy synced Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: federation state anti-entropy synced Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.714Z [TRACE] agent.server: federation state anti-entropy sync skipped; already up to date Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: federation state anti-entropy sync skipped; already up to date Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.730Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.930Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.938Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.serf.lan: serf: messageUserEventType: consul:new-leader Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.954Z [ERROR] agent.server.autopilot: Failed to reconcile current state with the desired state Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.autopilot: Failed to reconcile current state with the desired state Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.954Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.966Z [DEBUG] agent.server.cert-manager: server management token watch fired - resetting leaf cert watch Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: server management token watch fired - resetting leaf cert watch Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.967Z [DEBUG] agent.server.cert-manager: CA config watch fired - updating auto TLS server name: name=server.dc1.peering.d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: CA config watch fired - updating auto TLS server name: name=server.dc1.peering.d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.967Z [TRACE] agent.tlsutil: UpdateAutoTLSPeeringServerName: version=3 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.tlsutil: UpdateAutoTLSPeeringServerName: version=3 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.970Z [TRACE] agent.server: rpc_server_call: method=ConnectCA.Roots errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=ConnectCA.Roots errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.971Z [TRACE] connect.ca: authorizing and signing cert: spiffeID="&{d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul dc1}" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: connect.ca: authorizing and signing cert: spiffeID="&{d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul dc1}" Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.976Z [TRACE] agent.server: rpc_server_call: method=ConnectCA.Sign errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=ConnectCA.Sign errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=dc1 locality=local Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.976Z [DEBUG] agent.server.cert-manager: got cache update event: correlationID=leaf error= Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: got cache update event: correlationID=leaf error= Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.977Z [DEBUG] agent.server.cert-manager: leaf certificate watch fired - updating auto TLS certificate: uri=spiffe://d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul/agent/server/dc/dc1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.server.cert-manager: leaf certificate watch fired - updating auto TLS certificate: uri=spiffe://d103ce19-8e53-19ac-3c31-c4323bcfdeeb.consul/agent/server/dc/dc1 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:22.977Z [TRACE] agent.tlsutil: UpdateAutoTLSCert: version=4 Jun 07 14:24:22 ip-172-31-31-81 consul[1094]: agent.tlsutil: UpdateAutoTLSCert: version=4 Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.461Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'agent:read' on \"ip-172-31-31-81\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.462Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency="710.66µs" Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:58930 latency="710.66µs" Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.463Z [DEBUG] agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=dns Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.463Z [DEBUG] agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent: dropping service from result due to ACLs: service=consul Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.463Z [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency="748.949µs" Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:58930 latency="748.949µs" Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.464Z [ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent.http: Request error: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 error="Permission denied: anonymous token lacks permission 'service:write' on \"nomad\". The anonymous token is used implicitly when a request does not specify a token." Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:23.464Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="483.96µs" Jun 07 14:24:23 ip-172-31-31-81 consul[1094]: agent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:58930 latency="483.96µs" Jun 07 14:24:24 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:24.881Z [TRACE] agent.server: rpc_server_call: method=Catalog.ServiceNodes errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:24 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Catalog.ServiceNodes errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:24 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:24.954Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true Jun 07 14:24:24 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.242Z [TRACE] agent.server: rpc_server_call: method=Catalog.NodeServiceList errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Catalog.NodeServiceList errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.270Z [TRACE] agent.server: rpc_server_call: method=Catalog.ListDatacenters errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Catalog.ListDatacenters errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.271Z [TRACE] agent.server: rpc_server_call: method=Catalog.ServiceNodes errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=Catalog.ServiceNodes errored=true request_type=read rpc_type=net/rpc leader=true allow_stale=true blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.349Z [WARN] agent.server.acl: failed to remove bootstrap file: error="remove /opt/consul/data/acl-bootstrap-reset: no such file or directory" Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server.acl: failed to remove bootstrap file: error="remove /opt/consul/data/acl-bootstrap-reset: no such file or directory" Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.352Z [INFO] agent.server.acl: ACL bootstrap completed Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server.acl: ACL bootstrap completed Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: 2024-06-07T14:24:25.352Z [TRACE] agent.server: rpc_server_call: method=ACL.BootstrapTokens errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=dc1 locality=local Jun 07 14:24:25 ip-172-31-31-81 consul[1094]: agent.server: rpc_server_call: method=ACL.BootstrapTokens errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=dc1 locality=local ```
EugenKon commented 5 months ago

I resolved the problem. It was a problem with a host provision. The Consul was started with default config, then a new config was copied, then the consul was restarted like here.

It is better to not start the service ahead of time. https://github.com/hashicorp/learn-nomad-cluster-setup/blob/main/shared/scripts/server.sh#L48

Then I would not experience the problem later. https://github.com/hashicorp/learn-nomad-cluster-setup/blob/main/shared/data-scripts/user-data-client.sh#L16

So this is not the problem with consul, just a rough tutorial. Otherwise everything is good.