cloudfoundry-community / safe-boshrelease

A simplified HA Vault intended to be used with `safe`
MIT License
8 stars 9 forks source link

Vault/Consul unstable with stemcell 621.23 #30

Open hfrenzel opened 4 years ago

hfrenzel commented 4 years ago

After stemcell update to use 621.23, the safe deployment (v0.2.1) with vault/consul became very unstable as of following errors:

==> /var/vcap/sys/log/vault/consul.log <==
    2019/11/19 18:43:06 [WARN] raft: Election timeout reached, restarting election
    2019/11/19 18:43:06 [INFO] raft: Node at 10.X.X.X:8300 [Candidate] entering Candidate state in term 11581
    2019/11/19 18:43:07 [ERR] http: Request GET /v1/kv/vault/core/cluster/local/info, error: No cluster leader from=127.0.0.1:52548
    2019/11/19 18:43:10 [ERR] agent: coordinate update error: No cluster leader
    2019/11/19 18:43:10 [ERR] http: Request PUT /v1/session/renew/4469ca4a-7b14-adbf-29e7-b95b06c0af20, error: No cluster leader from=127.0.0.1:52578
    2019/11/19 18:43:11 [ERR] http: Request GET /v1/kv/vault/core/lock, error: No cluster leader from=127.0.0.1:52592
    2019/11/19 18:43:13 [WARN] raft: Election timeout reached, restarting election
    2019/11/19 18:43:13 [INFO] raft: Node at 10.X.X.X:8300 [Candidate] entering Candidate state in term 11582
    2019/11/19 18:43:16 [ERR] agent: failed to sync remote state: No cluster leader
    2019/11/19 18:43:16 [ERR] http: Request GET /v1/kv/vault/core/poison-pill, error: No cluster leader from=127.0.0.1:52594
    2019/11/19 18:43:18 [ERR] http: Request GET /v1/kv/vault/core/lock, error: No cluster leader from=127.0.0.1:52654
    2019/11/19 18:43:18 [ERR] http: Request GET /v1/kv/vault/core/?keys=&separator=%2F, error: No cluster leader from=127.0.0.1:52658

==> /var/vcap/sys/log/vault/vault.log <==
2019-11-19T18:43:18.626Z [ERROR] core: barrier init check failed: error="failed to check for initialization: Unexpected response code: 500"
2019-11-19T18:43:18.626Z [ERROR] core: error checking health: error="failed to check for initialization: Unexpected response code: 500"

After rolling back to the last used stemcell (621.12), the error 500 seems to be gone.