Next steps after terraform apply

arvindamirtaa commented 3 years ago

Question I think I'm missing a step. I cloned the repo, set the vars, ran terraform init, terraform plan and then terraform apply.

I got Apply complete! Resources: 41 added, 0 changed, 0 destroyed..

Then, I ran the root_token_decrypt_command and got the token. Set VAULT_ADDR, VAULT_TOKEN & VAULT_CAPATH as pert he README.md.

Assuming this is all correct, I should be able to interact with vault. Except I get...

Error enabling: Post "https://<IP>/v1/sys/mounts/secret": dial tcp <IP>:443: connect: connection refused

Upon digging a little further, the load balancer is reporting that the health check is failing for all 3 instances.

Considerations Is there a step I'm missing? Any pointers on what I can look into next?

agates4 commented 3 years ago

this is happening for me too: the load balancer is reporting that the health check is failing for all 3 instances.

@sethvargo looks like something happened where this terraform config no longer works out of the box - needs an update, this issue is reproducible

agates4 commented 3 years ago

vault secrets enable -path=secret -version=2 kv
Error enabling: Post "https://<IP>/v1/sys/mounts/secret": EOF

I actually get this error rather than a connection refused error

agates4 commented 3 years ago

all the pods are healthy upon Kubernetes inspection, it is just a failure of the health check (and also a failure of accessing the UI through port 8200)

agates4 commented 3 years ago

logs for vault-0 vault-init:

2021/08/10 23:13:17 Starting the vault-init service...
2021/08/10 23:13:17 Head http://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused
2021/08/10 23:13:27 Vault is not initialized.
2021/08/10 23:13:27 Initializing...
2021/08/10 23:13:30 Encrypting unseal keys and the root token...
2021/08/10 23:13:30 Unseal keys written to gs://<censored>-vault-storage/unseal-keys.json.enc
2021/08/10 23:13:30 Root token written to gs://<censored>-vault-storage/root-token.enc
2021/08/10 23:13:30 Initialization complete.
2021/08/10 23:13:30 Next check in 10s
2021/08/10 23:13:40 Vault is initialized and unsealed.
2021/08/10 23:13:40 Next check in 10s
2021/08/10 23:13:51 Vault is initialized and unsealed.
2021/08/10 23:13:51 Next check in 10s
2021/08/10 23:14:01 Vault is initialized and unsealed.
2021/08/10 23:14:01 Next check in 10s
2021/08/10 23:14:11 Vault is initialized and unsealed.

logs for vault-0 vault

==> Vault server configuration:

      GCP KMS Crypto Key: vault-init
        GCP KMS Key Ring:  <censored>
         GCP KMS Project: <censored>
          GCP KMS Region:  <censored>
               Seal Type: gcpckms
             Api Address:  <censored>
                     Cgo: disabled
         Cluster Address: https://10.0.92.5:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
              Listener 2: tcp (addr: "10.0.92.5:8200", cluster address: "10.0.92.5:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: warn
                   Mlock: supported: true, enabled: true
                 Storage: gcs (HA available)
                 Version: Vault v1.2.1

==> Vault server started! Log data will stream in below:

2021-08-10T23:13:24.383Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2021-08-10T23:13:27.389Z [WARN]  core: stored keys supported on init, forcing shares/threshold to 1
2021-08-10T23:13:30.637Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed

agates4 commented 3 years ago

this project is no longer supported ? @sethvargo

agates4 commented 3 years ago

on further inspection:

I reverted this PR: https://github.com/sethvargo/vault-on-gke/pull/94 and then everything started working as normal.

cheers!

bchrobot commented 3 years ago

PR #94 adds support for Kubernetes Service Registration. This project is still using Vault 1.2.1 (now 2 years old) by default, however, and Kubernetes Service Registration wasn't added until Vault 1.4. Vault is now at v1.8.1. Try updating the Vault version to 1.8.1 on sethvargo/vault-on-gke#master.

From the logs you posted, @agates4, it seems like there may be a separate Vault error unrelated to the updating of Pod labels

2021-08-10T23:13:24.383Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2021-08-10T23:13:27.389Z [WARN]  core: stored keys supported on init, forcing shares/threshold to 1
2021-08-10T23:13:30.637Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed

agates4 commented 3 years ago

thank you @bchrobot !

using these values

variable "vault_container" {
  type        = string
  default     = "vault:1.8.1"
  description = "Name of the Vault container image to deploy. This can be specified like \"container:version\" or as a full container URL."
}

variable "vault_init_container" {
  type        = string
  default     = "sethvargo/vault-init:0.2.0"
  description = "Name of the Vault init container image to deploy. This can be specified like \"container:version\" or as a full container URL."
}

within variables.tf, the up-to-date code works as expected. Cheers 🥂

sethvargo / vault-on-gke

Next steps after terraform apply #95