x509: certificate signed by unknown authority when trying to unseal vault

[AlexMussell]

Probably an extension of a previous issue (#153), however I am unsure if it will be being watched by the person who solved it.

After deployment, i need to manually and initialise the vault using -tls-skip-verify. Here is the vault job deployment:

The job:

jobs:
  - name: core
    properties:  
       vault:
          tls:
            ca:          ((vault-tls.ca))
            certificate: ((vault-tls.certificate))
            key:         ((vault-tls.private_key))

The variables:

variables: 
 - name: vault-ca
    type: certificate
    options:
      is_ca: true
      common_name: vaultca

  - name: vault-tls
    type: certificate
    options:
      ca: vault-ca
      common_name: vault
      extended_key_usage:
        - client_auth
        - server_auth
      alternative_names:
        - 127.0.0.1
        - "*.vault.default.shield.bosh"

I shouldn't need to update the release to accommodate for the flag. Leading me to believe that I am doing something wrong. This issue seems to reappear and leads to the same issue as (#153). That issue appears after unsealing the vault with ./vault unseal -tls-skip-verify. Any advice on either issue would be appreciated.

[AlexMussell]

Further update, it seems that the certificate store is not being updated with the ((vault-ca.ca)). As when I run

curl https://127.0.0.1:8200 --cacert vault.ca

I get a 404 not found. Without forcing curl to accept the ca, you get the standard:

curl: (60) SSL certificate problem: self signed certificate in certificate chain

On a slightly separate note (but still a possible), there is also an issue that the intermediate certificates are not being sent in the certificate chain .

Certificate chain
 0 s:/C=USA/O=Cloud Foundry/CN=vault
   i:/C=USA/O=Cloud Foundry/CN=vaultca
-----BEGIN CERTIFICATE-----
((vault-tls.certificate))
-----END CERTIFICATE-----
---
Server certificate
subject=/C=USA/O=Cloud Foundry/CN=vault
issuer=/C=USA/O=Cloud Foundry/CN=vaultca
---
No client certificate CA names sent
---
SSL handshake has read 1507 bytes and written 433 bytes
---

So I manually had to append it to the public key which returned the correct ssl profile:

---
Certificate chain
 0 s:/C=USA/O=Cloud Foundry/CN=vault
   i:/C=USA/O=Cloud Foundry/CN=vaultca
-----BEGIN CERTIFICATE-----
((vault-tls.certificate))
-----END CERTIFICATE-----
 1 s:/C=USA/O=Cloud Foundry/CN=vaultca
   i:/C=USA/O=Cloud Foundry/CN=vaultca
-----BEGIN CERTIFICATE-----
((vault-ca.ca))
-----END CERTIFICATE-----
---
Server certificate
subject=/C=USA/O=Cloud Foundry/CN=vault
issuer=/C=USA/O=Cloud Foundry/CN=vaultca
---
No client certificate CA names sent
---

Considering vault is only listening on the loopback interface, it won't be making external calls. So I just want to know if you knew how to get the certs to be trusted/why the ((vault-ca.ca)) isn't being added to the certificate store automatically, considering the v8.0.16 manifest hasn't been changed by me manually or via an operators file.

[AlexMussell]

To further extend (and possibly reslove), it doesn't seem like the ((vault.ca)) is actually used anywhere in the release. I had to manually create an environment variable on the vm: export VAULT_CACERT=/var/vcap/jobs/core/config/tls/vault.ca Then I was able to manually initialise the vault and get the keys without skipping tls verification.

If you could confirm that this is the correct behaviour, that would be great. If it is, it was not very clear. I assume you do it manually as to get the unseal keys.

Thanks, Alex

[jhunt]

The Vault listens on loopback because it is 100% managed by the SHIELD Core; for all intents and purposes, Vault is part of SHIELD and should not need any operator involvement / intervention except in extreme circumstances.

When Vault is sealed, the SHIELD Core recognizes this and reports itself as "locked"; unlocking a SHIELD Core involves supplying the SHIELD Master Password, at which point the following occurs:

1. An encryption key is derived form the master password (the "master key")
2. An encrypted on-disk file is decrypted with this master key
3. The plaintext contents of the encrypted file contain the seal keys for the Vault
4. SHIELD unseals the Vault
5. SHIELD forgets the seal keys

For brand new SHIELD installations, the Vault comes up as uninitialized. The SHIELD Core recognizes this and reports itself as "uninitialized". Initializing the SHIELD Core involves supplying a new master password, at which point the following occurs:

1. An encryption key is derived form the master password (the "master key")
2. SHIELD initializes the Vault, and receives the initial root token and seal keys
3. The initial root token and seal keys are encrypted with the master key
4. The encrypted data is written to disk
5. SHIELD forgets the seal keys

The Vault CA is used by the SHIELD Core to validate the certificate presented by the Vault when SHIELD connects to it over loopback. Essentially, this is a single-cert cert store / pool. The path to the Vault CA PEM file is placed in shieldd.conf, under the (8.0) vault_ca_cert key, here:

https://github.com/starkandwayne/shield-boshrelease/blob/c2659cf7a4ca4eeb46b44fbe38fe766bc5241990/jobs/core/templates/config/shieldd.conf#L35

Hope that clears a few things up 😄