search

nunchistudio / hashibox

Simulate a highly-available Cloud Platform with Consul, Nomad, and Vault. OSS & Enterprise versions supported.
https://nunchi.studio/hashibox
MIT License
88 stars 18 forks source link

Constraint "${attr.consul.version} semver >= 1.7.0": 1 nodes excluded by filter #4

Closed makp0 closed 1 year ago

makp0 commented 1 year ago

What OS are you running?

Linux

What architecture are you running?

amd64

What Vagrant provider are you running?

parallels

Unexpected behavior

When trying to run the Consul service mesh dashboard example from the official Hashi docs, I'm getting:

image

Steps to reproduce

Hashibox on commit: https://github.com/nunchistudio/hashibox/commit/0d3fc700e7a35829e668bb7747d964b629f4c0af

  1. run make init
  2. run the dashboard example job from Nomad UI

Expected behavior

No warning should appear

Logs

Apr 02 08:19:20 node-server-1 systemd[1]: Starting "HashiCorp Consul"... ░░ Subject: A start job for unit consul.service has begun execution ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A start job for unit consul.service has begun execution. ░░ ░░ The job identifier is 3287. Apr 02 08:19:20 node-server-1 consul[3692]: ==> Starting Consul agent... Apr 02 08:19:20 node-server-1 consul[3692]: Version: '1.15.2' Apr 02 08:19:20 node-server-1 consul[3692]: Build Date: '2023-03-30 17:51:19 +0000 UTC' Apr 02 08:19:20 node-server-1 consul[3692]: Node ID: 'a2dbd0ce-3500-2a9b-4624-2ddf766082c9' Apr 02 08:19:20 node-server-1 consul[3692]: Node name: 'node-server-1' Apr 02 08:19:20 node-server-1 consul[3692]: Datacenter: 'us' (Segment: '') Apr 02 08:19:20 node-server-1 consul[3692]: Server: true (Bootstrap: false) Apr 02 08:19:20 node-server-1 consul[3692]: Client Addr: [192.168.60.10] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600) Apr 02 08:19:20 node-server-1 consul[3692]: Cluster Addr: 192.168.60.10 (LAN: 8301, WAN: 8302) Apr 02 08:19:20 node-server-1 consul[3692]: Gossip Encryption: false Apr 02 08:19:20 node-server-1 consul[3692]: Auto-Encrypt-TLS: false Apr 02 08:19:20 node-server-1 consul[3692]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2 Apr 02 08:19:20 node-server-1 consul[3692]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2 Apr 02 08:19:20 node-server-1 consul[3692]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2 Apr 02 08:19:20 node-server-1 consul[3692]: ==> Log data will now stream in as it occurs: Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.941-0500 [WARN] agent: bootstrap_expect > 0: expecting 3 servers Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.949-0500 [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.raft: initial configuration: index=0 servers=[] Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-1.us 192.168.60.10 Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-1 192.168.60.10 Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.955-0500 [INFO] agent.router: Initializing LAN area manager Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.956-0500 [INFO] agent.server.autopilot: reconciliation now disabled Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.957-0500 [INFO] agent.server.raft: entering follower state: follower="Node at 192.168.60.10:8300 [Follower]" leader-address= leader-id= Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.959-0500 [INFO] agent.server: Adding LAN server: server="node-server-1 (Addr: tcp/192.168.60.10:8300) (DC: us)" Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.960-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-1.us area=wan Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.960-0500 [WARN] agent: [core][Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to { Apr 02 08:19:20 node-server-1 consul[3692]: "Addr": "us-192.168.60.10:8300", Apr 02 08:19:20 node-server-1 consul[3692]: "ServerName": "node-server-1", Apr 02 08:19:20 node-server-1 consul[3692]: "Attributes": null, Apr 02 08:19:20 node-server-1 consul[3692]: "BalancerAttributes": null, Apr 02 08:19:20 node-server-1 consul[3692]: "Type": 0, Apr 02 08:19:20 node-server-1 consul[3692]: "Metadata": null Apr 02 08:19:20 node-server-1 consul[3692]: }. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.60.10:0->192.168.60.10:8300: operation was canceled" Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.961-0500 [INFO] agent.server.cert-manager: initialized server certificate management Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=udp Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=tcp Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.967-0500 [INFO] agent: Starting server: address=192.168.60.10:8500 network=tcp protocol=http Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: started state syncer Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Consul agent running! Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Started gRPC listeners: port_name=grpc address=192.168.60.10:8502 network=tcp Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=192.168.60.10:8503 network=tcp Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: Joining cluster...: cluster=LAN Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.968-0500 [INFO] agent: (LAN) joining: lan_addresses=["192.168.60.10", "192.168.60.20", "192.168.60.30"] Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-3 192.168.60.30 Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-2 192.168.60.20 Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.992-0500 [INFO] agent.server: Adding LAN server: server="node-server-3 (Addr: tcp/192.168.60.30:8300) (DC: us)" Apr 02 08:19:20 node-server-1 consul[3692]: 2023-04-02T08:19:20.994-0500 [INFO] agent: (LAN) joined: number_of_nodes=3 Apr 02 08:19:20 node-server-1 systemd[1]: Started "HashiCorp Consul". ░░ Subject: A start job for unit consul.service has finished successfully ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A start job for unit consul.service has finished successfully. ░░ ░░ The job identifier is 3287. Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.017-0500 [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.028-0500 [INFO] agent.server: Found expected number of peers, attempting bootstrap: peers="192.168.60.30:8300,192.168.60.20:8300,192.168.60.10:8300" Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-3.us 192.168.60.30 Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-2.us 192.168.60.20 Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-3.us area=wan Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.048-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-2.us area=wan Apr 02 08:19:21 node-server-1 consul[3692]: 2023-04-02T08:19:21.061-0500 [INFO] agent.server: Adding LAN server: server="node-server-2 (Addr: tcp/192.168.60.20:8300) (DC: us)" Apr 02 08:19:28 node-server-1 consul[3692]: 2023-04-02T08:19:28.623-0500 [INFO] agent.server: New leader elected: payload=node-server-3 Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.665-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-1 192.168.61.10 Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.790-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-3 192.168.61.30 Apr 02 08:19:32 node-server-1 consul[3692]: 2023-04-02T08:19:32.808-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-2 192.168.61.20 Apr 02 08:19:33 node-server-1 consul[3692]: 2023-04-02T08:19:33.957-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:34 node-server-1 consul[3692]: 2023-04-02T08:19:34.960-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:35 node-server-1 consul[3692]: 2023-04-02T08:19:35.940-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:36 node-server-1 consul[3692]: 2023-04-02T08:19:36.522-0500 [ERROR] agent: Coordinate update error: error="rpc error making call: ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:36 node-server-1 consul[3692]: 2023-04-02T08:19:36.959-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:37 node-server-1 consul[3692]: 2023-04-02T08:19:37.451-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:39 node-server-1 consul[3692]: 2023-04-02T08:19:39.959-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:40 node-server-1 consul[3692]: 2023-04-02T08:19:40.963-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:41 node-server-1 consul[3692]: 2023-04-02T08:19:41.963-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:42 node-server-1 consul[3692]: 2023-04-02T08:19:42.965-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:44 node-server-1 consul[3692]: 2023-04-02T08:19:44.958-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Caught: signal=interrupt Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Graceful shutdown disabled. Exiting Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent: Requesting shutdown Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [INFO] agent.server: shutting down server Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.054-0500 [WARN] agent.server.serf.lan: serf: Shutdown without a Leave Apr 02 08:19:46 node-server-1 systemd[1]: Stopping "HashiCorp Consul"... ░░ Subject: A stop job for unit consul.service has begun execution ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A stop job for unit consul.service has begun execution. ░░ ░░ The job identifier is 4336. Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [WARN] agent.server.serf.wan: serf: Shutdown without a Leave Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [INFO] agent.router.manager: shutting down Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.056-0500 [INFO] agent.router.manager: shutting down Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: consul server down Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: shutdown complete Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: protocol=DNS address=192.168.60.10:8600 network=tcp Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: protocol=DNS address=192.168.60.10:8600 network=udp Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Stopping server: address=192.168.60.10:8500 network=tcp protocol=http Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Waiting for endpoints to shut down Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Endpoints down Apr 02 08:19:46 node-server-1 consul[3692]: 2023-04-02T08:19:46.057-0500 [INFO] agent: Exit code: code=1 Apr 02 08:19:46 node-server-1 systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ An ExecStart= process belonging to unit consul.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. Apr 02 08:19:46 node-server-1 systemd[1]: consul.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ The unit consul.service has entered the 'failed' state with result 'exit-code'. Apr 02 08:19:46 node-server-1 systemd[1]: Stopped "HashiCorp Consul". ░░ Subject: A stop job for unit consul.service has finished ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A stop job for unit consul.service has finished. ░░ ░░ The job identifier is 4336 and the job result is done. Apr 02 08:19:50 node-server-1 systemd[1]: Starting "HashiCorp Consul"... ░░ Subject: A start job for unit consul.service has begun execution ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A start job for unit consul.service has begun execution. ░░ ░░ The job identifier is 4336. Apr 02 08:19:50 node-server-1 consul[3822]: ==> Starting Consul agent... Apr 02 08:19:50 node-server-1 consul[3822]: Version: '1.15.2' Apr 02 08:19:50 node-server-1 consul[3822]: Build Date: '2023-03-30 17:51:19 +0000 UTC' Apr 02 08:19:50 node-server-1 consul[3822]: Node ID: 'a2dbd0ce-3500-2a9b-4624-2ddf766082c9' Apr 02 08:19:50 node-server-1 consul[3822]: Node name: 'node-server-1' Apr 02 08:19:50 node-server-1 consul[3822]: Datacenter: 'us' (Segment: '') Apr 02 08:19:50 node-server-1 consul[3822]: Server: true (Bootstrap: false) Apr 02 08:19:50 node-server-1 consul[3822]: Client Addr: [192.168.60.10] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600) Apr 02 08:19:50 node-server-1 consul[3822]: Cluster Addr: 192.168.60.10 (LAN: 8301, WAN: 8302) Apr 02 08:19:50 node-server-1 consul[3822]: Gossip Encryption: false Apr 02 08:19:50 node-server-1 consul[3822]: Auto-Encrypt-TLS: false Apr 02 08:19:50 node-server-1 consul[3822]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2 Apr 02 08:19:50 node-server-1 consul[3822]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2 Apr 02 08:19:50 node-server-1 consul[3822]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2 Apr 02 08:19:50 node-server-1 consul[3822]: ==> Log data will now stream in as it occurs: Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.548-0500 [WARN] agent: bootstrap_expect > 0: expecting 3 servers Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.551-0500 [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:247b43d4-680e-91a7-2559-ed5f7edf04f0 Address:192.168.60.30:8300} {Suffrage:Voter ID:0568fec4-01e3-982f-d971-9954e2fe12c6 Address:192.168.60.20:8300} {Suffrage:Voter ID:a2dbd0ce-3500-2a9b-4624-2ddf766082c9 Address:192.168.60.10:8300}]" Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-1.us 192.168.60.10 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-1 192.168.60.10 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.554-0500 [INFO] agent.router: Initializing LAN area manager Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.autopilot: reconciliation now disabled Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.raft: entering follower state: follower="Node at 192.168.60.10:8300 [Follower]" leader-address= leader-id= Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: node-server-3.us: 192.168.60.30:8302 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: node-server-2: 192.168.60.20:8301 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.555-0500 [INFO] agent.server: Adding LAN server: server="node-server-1 (Addr: tcp/192.168.60.10:8300) (DC: us)" Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.556-0500 [INFO] agent.server: Raft data found, disabling bootstrap mode Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.556-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-1.us area=wan Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.cert-manager: initialized server certificate management Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=udp Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-2 192.168.61.20 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-3 192.168.60.30 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-server-2 192.168.60.20 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: node-server-2: 192.168.60.20:8301 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server: Adding LAN server: server="node-server-3 (Addr: tcp/192.168.60.30:8300) (DC: us)" Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent.server: Adding LAN server: server="node-server-2 (Addr: tcp/192.168.60.20:8300) (DC: us)" Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Started DNS server: address=192.168.60.10:8600 network=tcp Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.557-0500 [INFO] agent: Starting server: address=192.168.60.10:8500 network=tcp protocol=http Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: started state syncer Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Consul agent running! Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Started gRPC listeners: port_name=grpc address=192.168.60.10:8502 network=tcp Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=192.168.60.10:8503 network=tcp Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: Joining cluster...: cluster=LAN Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.558-0500 [INFO] agent: (LAN) joining: lan_addresses=["192.168.60.10", "192.168.60.20", "192.168.60.30"] Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.559-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-2.us 192.168.60.20 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.559-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-2.us area=wan Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.564-0500 [INFO] agent.server.serf.wan: serf: EventMemberJoin: node-server-3.us 192.168.60.30 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.564-0500 [INFO] agent.server: Handled event for server in area: event=member-join server=node-server-3.us area=wan Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.566-0500 [INFO] agent: (LAN) joined: number_of_nodes=3 Apr 02 08:19:50 node-server-1 systemd[1]: Started "HashiCorp Consul". ░░ Subject: A start job for unit consul.service has finished successfully ░░ Defined-By: systemd ░░ Support: http://www.ubuntu.com/support ░░ ░░ A start job for unit consul.service has finished successfully. ░░ ░░ The job identifier is 4336. Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.587-0500 [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: node-server-3.us: 192.168.60.30:8302 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.587-0500 [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.634-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-3 192.168.61.30 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.651-0500 [INFO] agent.server.serf.lan: serf: EventMemberJoin: node-client-1 192.168.61.10 Apr 02 08:19:50 node-server-1 consul[3822]: 2023-04-02T08:19:50.663-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:51 node-server-1 consul[3822]: 2023-04-02T08:19:51.670-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:53 node-server-1 consul[3822]: 2023-04-02T08:19:53.569-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:54 node-server-1 consul[3822]: 2023-04-02T08:19:54.575-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:55 node-server-1 consul[3822]: 2023-04-02T08:19:55.679-0500 [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.60.10:48840 error="ACL system must be bootstrapped before making any requests that require authorization: ACL not found" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.559-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.583-0500 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.585-0500 [INFO] agent.server.raft: entering candidate state: node="Node at 192.168.60.10:8300 [Candidate]" term=3 Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: election won: term=3 tally=2 Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: entering leader state: leader="Node at 192.168.60.10:8300 [Leader]" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.611-0500 [INFO] agent.server.raft: added peer, starting replication: peer=247b43d4-680e-91a7-2559-ed5f7edf04f0 Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server.raft: added peer, starting replication: peer=0568fec4-01e3-982f-d971-9954e2fe12c6 Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server: cluster leadership acquired Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.612-0500 [INFO] agent.server: New leader elected: payload=node-server-1 Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.613-0500 [INFO] agent.server.raft: pipelining replication: peer="{Voter 0568fec4-01e3-982f-d971-9954e2fe12c6 192.168.60.20:8300}" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.614-0500 [INFO] agent.server.raft: pipelining replication: peer="{Voter 247b43d4-680e-91a7-2559-ed5f7edf04f0 192.168.60.30:8300}" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.619-0500 [INFO] agent.server: initializing acls Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.leader: started routine: routine="acl token reaping" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.server.autopilot: reconciliation now enabled Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.623-0500 [INFO] agent.leader: started routine: routine="federation state anti-entropy" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="federation state pruning" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="streaming peering resources" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="metrics for streaming peering resources" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="peering deferred deletion" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] connect.ca: initialized primary datacenter CA from existing CARoot with provider: provider=consul Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="intermediate cert renew watch" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.624-0500 [INFO] agent.leader: started routine: routine="CA root pruning" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="CA root expiration metric" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="CA signing expiration metric" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="virtual IP version check" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.625-0500 [INFO] agent.leader: started routine: routine="config entry controllers" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.626-0500 [INFO] agent.leader: stopping routine: routine="virtual IP version check" Apr 02 08:19:56 node-server-1 consul[3822]: 2023-04-02T08:19:56.626-0500 [INFO] agent.leader: stopped routine: routine="virtual IP version check" Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.300-0500 [WARN] agent.server.acl: failed to remove bootstrap file: error="remove /opt/consul/acl-bootstrap-reset: no such file or directory" Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.329-0500 [INFO] agent.server.acl: ACL bootstrap completed Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.365-0500 [INFO] agent.http: Request cancelled: method=GET url="/v1/catalog/service/nomad?dc=us&near=_agent&stale=&tag=serf&wait=2000ms" from=192.168.60.10:48842 error="ACL not found" Apr 02 08:19:57 node-server-1 consul[3822]: 2023-04-02T08:19:57.560-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-2 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:58 node-server-1 consul[3822]: 2023-04-02T08:19:58.560-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-1 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:19:59 node-server-1 consul[3822]: 2023-04-02T08:19:59.563-0500 [WARN] agent.server.memberlist.lan: memberlist: Was able to connect to node-client-3 over TCP but UDP probes failed, network may be misconfigured Apr 02 08:20:02 node-server-1 consul[3822]: 2023-04-02T08:20:02.425-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found" Apr 02 08:20:02 node-server-1 consul[3822]: 2023-04-02T08:20:02.711-0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"

loicsaintroch commented 1 year ago

Given the logs of the Consul Client, it looks like this is related to #3, right?

makp0 commented 1 year ago

Oh my gosh, I somehow managed to miss the single issue in this repo. Hilarious! Yes, it's definitely a duplicate. Closing as duplicate.