search

binlab / terraform-aws-vault-ha-raft

Hashicorp Vault HA cluster based on Raft Consensus Algorithm
https://github.com/binlab/terraform-aws-vault-ha-raft
MIT License
14 stars 6 forks source link

Intermittent 502 errors #69

Open rhinck opened 3 years ago

rhinck commented 3 years ago

I'm running the latest version of the module from the main branch and I've been having an issue over the last few weeks where loading the UI or Vault from client libraries will intermittently receive a 502 gateway error.

Do you have an idea of what could potentially be causing that and where I can look to try and get logs?

binlab commented 3 years ago

I would start with looking at AWS metrics for that period of time for the LoadBalancer and EC2 (CPU, how many requests, how many errors etc). Then connecting to the instance via SSH and looking a logs. Some useful commands for viewing logs are described here

rhinck commented 3 years ago

Here's a graph from ELB of how many 502 errors I've been getting over the last 3 days:

Screen Shot 2021-09-02 at 9 25 02 AM
rhinck commented 3 years ago

Here are the logs I get when I run this command: journalctl --utc -a -u vault.service -f

-- Journal begins at Thu 2021-09-02 17:52:47 UTC. --
Sep 02 18:51:43 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:51:43.021Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=node2.vault.int:8201
Sep 02 18:51:43 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:51:43.021Z [INFO]  storage.raft: entering candidate state: node="Node at node0.vault.int:8201 [Candidate]" term=49
Sep 02 18:51:43 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:51:43.025Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter node2 node2.vault.int:8201}" error="dial tcp 192.168.3.74:8201: connect: connection refused"
Sep 02 18:51:46 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:51:46.195Z [INFO]  storage.raft: duplicate requestVote for same term: term=49
Sep 02 18:51:46 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:51:46.328Z [INFO]  storage.raft: entering follower state: follower="Node at node0.vault.int:8201 [Follower]" leader=node1.vault.int:8201
Sep 02 18:52:51 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:52:51.789Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=node1.vault.int:8201
Sep 02 18:52:51 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:52:51.789Z [INFO]  storage.raft: entering candidate state: node="Node at node0.vault.int:8201 [Candidate]" term=50
Sep 02 18:52:51 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:52:51.791Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter node1 node1.vault.int:8201}" error=EOF
Sep 02 18:52:58 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:52:58.765Z [INFO]  storage.raft: duplicate requestVote for same term: term=50
Sep 02 18:52:58 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:52:58.776Z [INFO]  storage.raft: entering follower state: follower="Node at node0.vault.int:8201 [Follower]" leader=node1.vault.int:8201
Sep 02 18:54:03 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:03.907Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=node1.vault.int:8201
Sep 02 18:54:03 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:03.907Z [INFO]  storage.raft: entering candidate state: node="Node at node0.vault.int:8201 [Candidate]" term=51
Sep 02 18:54:03 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:03.910Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter node1 node1.vault.int:8201}" error="dial tcp 192.168.2.142:8201: connect: connection refused"
Sep 02 18:54:04 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:04.124Z [INFO]  storage.raft: duplicate requestVote for same term: term=51
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.776Z [WARN]  storage.raft: Election timeout reached, restarting election
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.776Z [INFO]  storage.raft: entering candidate state: node="Node at node0.vault.int:8201 [Candidate]" term=52
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.794Z [INFO]  storage.raft: election won: tally=2
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.794Z [INFO]  storage.raft: entering leader state: leader="Node at node0.vault.int:8201 [Leader]"
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.794Z [INFO]  storage.raft: added peer, starting replication: peer=node1
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.794Z [INFO]  storage.raft: added peer, starting replication: peer=node2
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.796Z [INFO]  storage.raft: pipelining replication: peer="{Voter node2 node2.vault.int:8201}"
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.814Z [INFO]  core: acquired lock, enabling active operation
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.891Z [INFO]  core: post-unseal setup starting
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.899Z [INFO]  core: loaded wrapping token key
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.899Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.900Z [INFO]  core: successfully mounted backend: type=system path=sys/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.900Z [INFO]  core: successfully mounted backend: type=identity path=identity/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.900Z [INFO]  core: successfully mounted backend: type=kv path=twilio/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.901Z [INFO]  core: successfully mounted backend: type=gcp path=gcp/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.901Z [INFO]  core: successfully mounted backend: type=kv path=environments/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.901Z [INFO]  core: successfully mounted backend: type=kv path=logins/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.901Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.903Z [INFO]  core: successfully enabled credential backend: type=token path=token/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.904Z [INFO]  core: successfully enabled credential backend: type=approle path=approle/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.904Z [INFO]  core: successfully enabled credential backend: type=github path=github/
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.905Z [INFO]  core: restoring leases
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.907Z [INFO]  rollback: starting rollback manager
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.915Z [INFO]  identity: entities restored
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.915Z [INFO]  identity: groups restored
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.915Z [INFO]  core: starting raft active node
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.915Z [INFO]  storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s}" reconcile_interval=0s
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.921Z [WARN]  core.raft: skipping new raft TLS config creation, keys are pending
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.972Z [INFO]  core: usage gauge collection is disabled
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.979Z [INFO]  core: post-unseal setup complete
Sep 02 18:54:09 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:09.990Z [INFO]  expiration: lease restore complete
Sep 02 18:54:10 ip-192-168-1-100.us-west-2.compute.internal docker[3442]: 2021-09-02T18:54:10.007Z [INFO]  storage.raft: pipelining replication: peer="{Voter node1 node1.vault.int:8201}"
rhinck commented 3 years ago

Here are some more for when I got the 502 error to occur again:


Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.470Z [INFO]  rollback: starting rollback manager
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.471Z [INFO]  core: restoring leases
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.473Z [INFO]  identity: entities restored
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.480Z [INFO]  identity: groups restored
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.480Z [INFO]  core: starting raft active node
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.480Z [INFO]  storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s}" reconcile_interval=0s
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.483Z [WARN]  core.raft: skipping new raft TLS config creation, keys are pending
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.522Z [INFO]  core: usage gauge collection is disabled
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.534Z [INFO]  core: post-unseal setup complete
Sep 02 21:34:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: 2021-09-02T21:34:00.546Z [INFO]  expiration: lease restore complete
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x3142c7a]
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: goroutine 1062 [running]:
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault-plugin-secrets-gcp/plugin.(*backend).serviceAccountPolicyRollback(0xc001214ff0, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0x47430a0, 0xc00137f350, 0x0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/pkg/mod/github.com/hashicorp/vault-plugin-secrets-gcp@v0.10.1/plugin/rollback.go:205 +0x15a
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault-plugin-secrets-gcp/plugin.(*backend).walRollback(0xc001214ff0, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0xc000e0bb50, 0xa, 0x47430a0, 0xc00137f350, 0xc00137f260, 0xc0011dd840)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/pkg/mod/github.com/hashicorp/vault-plugin-secrets-gcp@v0.10.1/plugin/rollback.go:30 +0x185
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/sdk/framework.(*Backend).handleWALRollback(0xc0006fd040, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0x45548f, 0xc000daccf0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/sdk/framework/backend.go:540 +0x4a8
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/sdk/framework.(*Backend).handleRollback(0xc0006fd040, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0x1c1a1692, 0x1c1a16920041455b, 0x61314384)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/sdk/framework/backend.go:483 +0xc8
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/sdk/framework.(*Backend).HandleRequest(0xc0006fd040, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0x0, 0x0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/sdk/framework/backend.go:192 +0xb51
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/builtin/plugin.(*PluginBackend).HandleRequest.func1(0x40dc1b, 0x5f469fe9977ee9ea)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/builtin/plugin/backend.go:199 +0x5a
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/builtin/plugin.(*PluginBackend).lazyLoadBackend(0xc00039f590, 0x5af2f58, 0xc000dba060, 0x5af48b8, 0xc001104a40, 0xc000f1f940, 0x0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/builtin/plugin/backend.go:162 +0x9a
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/builtin/plugin.(*PluginBackend).HandleRequest(0xc00039f590, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0x0, 0xc000dace00, 0x28)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/builtin/plugin/backend.go:197 +0xbb
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/vault.(*Router).routeCommon(0xc000bd4000, 0x5af2f58, 0xc000dba060, 0xc000d55340, 0xc000dba000, 0x0, 0x0, 0x0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/vault/router.go:686 +0x959
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/vault.(*Router).Route(...)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/vault/router.go:484
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: github.com/hashicorp/vault/vault.(*RollbackManager).attemptRollback(0xc000c5bdd0, 0x5af2f90, 0xc00137e120, 0xc0003ab528, 0x4, 0xc0010ffc00, 0x1, 0x0, 0x0)
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/vault/rollback.go:225 +0x3ae
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]: created by github.com/hashicorp/vault/vault.(*RollbackManager).startOrLookupRollback
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal docker[8265]:       /gopath/src/github.com/hashicorp/vault/vault/rollback.go:161 +0x254
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: vault.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 02 21:35:00 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: vault.service: Failed with result 'exit-code'.
Sep 02 21:35:05 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: vault.service: Scheduled restart job, restart counter is at 62.
Sep 02 21:35:05 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: Stopped "Hashicorp Vault HA".
Sep 02 21:35:05 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: Starting "Hashicorp Vault HA"...
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8382]: Error response from daemon: No such container: vault.service
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal systemd[1]: Started "Hashicorp Vault HA".
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: ==> Vault server configuration:
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:              Api Address: https://vault.lovage.dev:443
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:                      Cgo: disabled
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:          Cluster Address: https://node0.vault.int:8201
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:               Go Version: go1.16.6
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:                Log Level: info
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:                    Mlock: supported: true, enabled: true
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:            Recovery Mode: false
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:                  Storage: raft (HA available)
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:                  Version: Vault v1.8.1
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]:              Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: ==> Vault server started! Log dat2021-09-02T21:35:06.519Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.632Z [INFO]  core: stored unseal keys supported, attempting fetch
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.633Z [INFO]  core: raft retry join initiated
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: a will stream in below:
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.652Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.652Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.653Z [INFO]  storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:5000000000, ElectionTimeout:5000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:"node0", NotifyCh:(chan<- bool)(0xc000d72310), LogOutput:io.Writer(nil), LogLevel:"DEBUG", Logger:(*hclog.interceptLogger)(0xc0009777a0), NoSnapshotRestoreOnStart:true, skipStartup:false}"
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.655Z [INFO]  storage.raft: initial configuration: index=34 servers="[{Suffrage:Voter ID:node1 Address:node1.vault.int:8201} {Suffrage:Voter ID:node2 Address:node2.vault.int:8201} {Suffrage:Voter ID:node0 Address:node0.vault.int:8201}]"
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.655Z [INFO]  core: vault is unsealed
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.655Z [INFO]  core: unsealed with stored key
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.655Z [INFO]  core: entering standby mode
Sep 02 21:35:06 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:06.656Z [INFO]  storage.raft: entering follower state: follower="Node at node0.vault.int:8201 [Follower]" leader=
Sep 02 21:35:08 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:08.843Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
Sep 02 21:35:09 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:09.160Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
Sep 02 21:35:10 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:10.162Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
Sep 02 21:35:10 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:10.172Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
Sep 02 21:35:11 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:11.941Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
Sep 02 21:35:13 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:13.559Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=
Sep 02 21:35:13 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:13.559Z [INFO]  storage.raft: entering candidate state: node="Node at node0.vault.int:8201 [Candidate]" term=202
Sep 02 21:35:13 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:13.915Z [INFO]  storage.raft: entering follower state: follower="Node at node0.vault.int:8201 [Follower]" leader=
Sep 02 21:35:14 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:14.028Z [WARN]  storage.raft: skipping application of old log: index=150249
Sep 02 21:35:15 ip-192-168-1-100.us-west-2.compute.internal docker[8390]: 2021-09-02T21:35:15.633Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]```