trevop
commented
10 months ago Logs from service
==> Vault server configuration:
Administrative Namespace:
Api Address: http://hashicorp-vault-05.domain.dev:8200
Cgo: disabled
Cluster Address: https://hashicorp-vault-05.domain.dev:8201
Environment Variables: BASH_FUNC_which%%, DEBUGINFOD_URLS, GODEBUG, HISTSIZE, HOME, HOSTNAME, LANG, LC_ALL, LC_CTYPE, LESSOPEN, LOGNAME, LS_COLORS, MAIL, OLDPWD, PATH, PWD, SHELL, SHLVL, SUDO_COMMAND, SUDO_GID, SUDO_UID, SUDO_USER, S_COLORS, TERM, USER, VAULT_ADDR, VAULT_CLIENT_TIMEOUT, _, which_declare
Go Version: go1.21.3
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "hashicorp-vault-05.domain.dev:8201", max_request_duration: "3m0s", max_request_size: "33554432", tls: "disabled")
Log Level:
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.15.1, built 2023-10-20T19:16:11Z
Version Sha: b94e275f25ccd9011146d14c00ea9e49fd5032dc
==> Vault server started! Log data will stream in below:
2023-10-31T15:24:09.683Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-10-31T15:24:10.073Z [INFO] incrementing seal generation: generation=1
2023-10-31T15:24:10.074Z [INFO] core: Initializing version history cache for core
2023-10-31T15:24:10.074Z [INFO] events: Starting event system
2023-10-31T15:24:10.074Z [INFO] core: raft retry join initiated
2023-10-31T15:24:10.074Z [INFO] core: stored unseal keys supported, attempting fetch
2023-10-31T15:24:10.074Z [INFO] core: security barrier not initialized
2023-10-31T15:24:10.074Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-10-31T15:24:10.075Z [INFO] core: security barrier not initialized
2023-10-31T15:24:10.075Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://hashicorp-vault-03.domain.dev:8200
2023-10-31T15:24:10.075Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://hashicorp-vault-02.domain.dev:8200
2023-10-31T15:24:10.075Z [INFO] core: attempting to join possible raft leader node: leader_addr=http://hashicorp-vault-04.domain.dev:8200
2023-10-31T15:24:10.146Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=<node_ip>:8201
2023-10-31T15:24:10.146Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=<node_ip>:8201
2023-10-31T15:24:10.147Z [INFO] storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:15000000000, ElectionTimeout:15000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:\"hashicorp-vault-05\", NotifyCh:(chan<- bool)(0xc0034e3ea0), LogOutput:io.Writer(nil), LogLevel:\"DEBUG\", Logger:(*hclog.interceptLogger)(0xc002eba9f0), NoSnapshotRestoreOnStart:true, skipStartup:false}"
2023-10-31T15:24:10.148Z [INFO] storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:hashicorp-vault-03 Address:hashicorp-vault-03.domain.dev:8201} {Suffrage:Voter ID:hashicorp-vault-04 Address:hashicorp-vault-04.domain.dev:8201} {Suffrage:Voter ID:hashicorp-vault-02 Address:hashicorp-vault-02.domain.dev:8201} {Suffrage:Nonvoter ID:hashicorp-vault-05 Address:follower-sec-hashicorp-vault-hotel-05.domain.dev:8201}]"
2023-10-31T15:24:10.148Z [INFO] core: successfully joined the raft cluster: leader_addr=http://hashicorp-vault-04.domain.dev:8200
2023-10-31T15:24:10.148Z [INFO] storage.raft: entering follower state: follower="Node at hashicorp-vault-05.domain.dev:8201 [Follower]" leader-address= leader-id=
2023-10-31T15:24:15.075Z [INFO] core: stored unseal keys supported, attempting fetch
2023-10-31T15:24:15.075Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-10-31T15:24:20.075Z [INFO] core: stored unseal keys supported, attempting fetch
2023-10-31T15:24:20.075Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-10-31T15:24:25.076Z [INFO] core: stored unseal keys supported, attempting fetch
2023-10-31T15:24:25.076Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-10-31T15:24:26.628Z [WARN] storage.raft: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2023-10-31T15:24:30.077Z [INFO] core: stored unseal keys supported, attempting fetch
2023-10-31T15:24:30.077Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
and so on repeating failed to unseal core: error="stored unseal keys are supported, but none were found error
Fore-4
thnee
upenderadepu-moe
SanduDS
e100
Describe the bug Hi. I have a cluster of 3 nodes (initialized, unsealed, cluster is working and has a lot of secrets) with raft storage backend with auto-unseal. Auto-unseal type is GCP KMS.
I am trying to add 2 new nodes to the cluster (one by one). Config is delivered using ansible and is the same across the cluster (except for addr and other node to node params).
After node service is started I receive the following error
2023-10-31T15:31:55.144Z [INFO] core: stored unseal keys supported, attempting fetch2023-10-31T15:31:55.144Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"To Reproduce Steps to reproduce the behavior:
vault server -config vault.hclExpected behavior A new node successfully joins the cluster and is unsealed with auto-unseal
Environment:
vault status):1.15.1vault version):Vault v1.15.1 (b94e275f25ccd9011146d14c00ea9e49fd5032dc), built 2023-10-20T19:16:11ZCentOS 8 Stream x86_64Vault server configuration file(s):
Vault service systemd file: