vault in degraded state

I unsealed the vault, but login does not work

The only running pod opf-vault-0 logs:

2022-09-30T08:12:17.642Z [INFO]  storage.raft: entering candidate state: node="Node at opf-vault-0.opf-vault-internal:8201 [Candidate]" term=9984
2022-09-30T08:12:17.670Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 56796c78-adac-b69e-8a62-c431b17c8b9b opf-vault-2.opf-vault-internal:8201}" error="dial tcp 10.128.4.49:8201: connect: connection refused"
2022-09-30T08:12:17.670Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter e58ab8b5-62d7-cc6f-27e0-79656bfb8ab7 opf-vault-1.opf-vault-internal:8201}" error="dial tcp 10.131.2.39:8201: connect: connection refused"
2022-09-30T08:12:26.248Z [WARN]  storage.raft: Election timeout reached, restarting election

The other 2 pods are in a not ready state, logs:

==> Vault server configuration:

             Api Address: http://10.131.2.39:8200
                     Cgo: disabled
         Cluster Address: https://opf-vault-1.opf-vault-internal:8201
              Go Version: go1.17.5
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

==> Vault server started! Log data will stream in below:

2022-09-30T06:23:46.238Z [INFO]  proxy environment: http_proxy="\"\"" https_proxy="\"\"" no_proxy="\"\""
2022-09-30T06:23:46.321Z [WARN]  storage.raft.fsm: raft FSM db file has wider permissions than needed: needed=-rw------- existing=-rw-rw----
2022-09-30T06:24:16.360Z [INFO]  core: Initializing VersionTimestamps for core

the readiness probe fails with:

Readiness probe failed: Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.9.2 Storage Type raft HA Enabled true

/assign @HumairAK

operate-first / support

vault in degraded state #1107