hashicorp / vault-helm

Helm chart to install Vault and other associated components.
Mozilla Public License 2.0
1.07k stars 873 forks source link

Unable to unseal vault post init activity in Vault HA on K8s using S3 Storage #226

Open sudarshntn opened 4 years ago

sudarshntn commented 4 years ago

I have setup a 3 worker node Kubernetes Cluster and have setup S3 Storage with Bucket in it. I have used the Helm Chart and deployed with 3 replica's. The deployment and pod running status is all fine.

URL: PUT http://127.0.0.1:8200/v1/sys/unseal Code: 400. Errors:

Need urgent help to fix this issue.

More details are as below: Sudarshans-MBP vault-helm % kubectl get pods NAME READY STATUS RESTARTS AGE vault-0 0/1 Running 0 18s vault-1 0/1 Running 0 18s vault-2 0/1 Running 0 18s

Before init activity: Sudarshans-MBP vault-helm % kubectl exec -ti vault-0 sh / $ vault status Key Value


Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version n/a HA Enabled false

post init activity: / $ vault status Key Value


Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.3.3 HA Enabled false

vault-0 pod logs: 2020-03-12T05:56:12.484Z [INFO] core: security barrier not initialized 2020-03-12T05:56:15.483Z [INFO] core: security barrier not initialized 2020-03-12T05:56:18.484Z [INFO] core: security barrier not initialized 2020-03-12T05:56:21.485Z [INFO] core: security barrier not initialized 2020-03-12T05:56:24.483Z [INFO] core: security barrier not initialized 2020-03-12T05:56:26.729Z [INFO] core: seal configuration missing, not initialized 2020-03-12T05:56:27.483Z [INFO] core: security barrier not initialized 2020-03-12T05:56:30.483Z [INFO] core: security barrier not initialized 2020-03-12T05:56:33.539Z [INFO] core: security barrier not initialized 2020-03-12T05:56:35.204Z [ERROR] core: no seal config found, can't determine if legacy or new-style shamir 2020-03-12T05:56:35.276Z [INFO] core: security barrier not initialized 2020-03-12T05:56:35.646Z [INFO] core: security barrier initialized: stored=1 shares=5 threshold=3 2020-03-12T05:56:35.973Z [INFO] core: post-unseal setup starting 2020-03-12T05:56:36.070Z [INFO] core: loaded wrapping token key 2020-03-12T05:56:36.070Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2020-03-12T05:56:36.143Z [INFO] core: no mounts; adding default mount table 2020-03-12T05:56:36.238Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2020-03-12T05:56:36.239Z [INFO] core: successfully mounted backend: type=system path=sys/ 2020-03-12T05:56:36.239Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2020-03-12T05:56:37.036Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2020-03-12T05:56:37.036Z [INFO] core: restoring leases 2020-03-12T05:56:37.036Z [INFO] rollback: starting rollback manager 2020-03-12T05:56:37.119Z [INFO] expiration: lease restore complete 2020-03-12T05:56:37.382Z [INFO] identity: entities restored 2020-03-12T05:56:37.420Z [INFO] identity: groups restored 2020-03-12T05:56:37.573Z [INFO] core: post-unseal setup complete 2020-03-12T05:56:37.877Z [INFO] core: root token generated 2020-03-12T05:56:37.877Z [INFO] core: pre-seal teardown starting 2020-03-12T05:56:37.877Z [INFO] rollback: stopping rollback manager 2020-03-12T05:56:37.877Z [INFO] core: pre-seal teardown complete

hcl file content: cat vault/config/extraconfig-from-values.hcl disable_mlock = true ui = true

listener "tcp" { tls_disable = 1 address = "0.0.0.0:8200" cluster_address = "[::]:8201" } storage "s3" { access_key = "12345678999887765" secret_key = "098765445678854332" bucket = "suda-vault-test1-data" endpoint = "s3.cloud-object-storage" region = "us-east" path = "/data/vault" }

And I can view the data written to the storage.

sudarshntn commented 4 years ago

Post removing the path entry from the hcl file for S3 the Unseal worked. Is this a Bug in 1.3.3 version.

hcl file content: cat vault/config/extraconfig-from-values.hcl disable_mlock = true ui = true

listener "tcp" { tls_disable = 1 address = "0.0.0.0:8200" cluster_address = "[::]:8201" } storage "s3" { access_key = "12345678999887765" secret_key = "098765445678854332" bucket = "suda-vault-test1-data" endpoint = "s3.cloud-object-storage" region = "us-east" }

This config works. Need to get input on this as I saw a similar defect that was fixed in 1.3.3 version of Vault container.