post-start script pass but no changes, vault is sealed

[Alexvianet]

/var/vcap/jobs/vault/data/unseal_keys include 3 kays and one empty line

- azs:
  - AZ1
  - AZ1
  - AZ3
  instances: 3
  jobs:
  - name: consul
    release: consul
  - name: vault
    properties:
      tls:
      - cert: cert
      - key: key
        name: vault
      vault:
        config: |
          storage "consul" {
            path = "vault/"
            check_timeout = "5s"
            max_parallel = "128"
          }
          listener "tcp" {
            address = "0.0.0.0:8200"
            tls_cert_file = "/var/vcap/jobs/vault/tls/vault/cert.pem"
            tls_key_file  = "/var/vcap/jobs/vault/tls/vault/key.pem"
            tls_min_version = "tls12"
          }
          api_addr = "http://(ip):8200"
          cluster_addr = "https://(ip):8201"
        update:
          unseal_keys:
          - key1
          - key2
          - key3

deployment pass without failed but vault was not unseal

i try to run script by root

vault/f592421b-3aec-4cf1-915b-c6e19725e0c6:/var/vcap/jobs/vault/bin# ./post-start 
+ source /var/vcap/jobs/vault/helpers/ctl_setup.sh vault
++ set -e
++ set -u
++ JOB_NAME=vault
++ output_label=vault
++ export JOB_DIR=/var/vcap/jobs/vault
++ JOB_DIR=/var/vcap/jobs/vault
++ chmod 755 /var/vcap/jobs/vault
++ source /var/vcap/jobs/vault/data/properties.sh
+++ export NAME=vault
+++ NAME=vault
+++ export JOB_INDEX=2
+++ JOB_INDEX=2
+++ export JOB_FULL=vault/2
+++ JOB_FULL=vault/2
+++ export VAULT_ADDR=https://127.0.0.1:8200
+++ VAULT_ADDR=https://127.0.0.1:8200
+++ export VAULT_TOKEN=
+++ VAULT_TOKEN=
++ source /var/vcap/jobs/vault/helpers/ctl_utils.sh
++ redirect_output vault
++ SCRIPT=vault
++ mkdir -p /var/vcap/sys/log/monit
++ exec
++ exec
vault/f592421b-3aec-4cf1-915b-c6e19725e0c6:/var/vcap/jobs/vault/bin# 

[Alexvianet]

using latest stemcell bosh-vsphere-esxi-ubuntu-trusty-go_agent 3586.23* ubuntu-trusty - sc-d11422a1-19df-48d3-bf26-46abe88a9174

[Alexvianet]

@jhunt any updated, concerns?

[jhunt]

Does anything show up in the monit_debugger logs for the post-start script?

[Alexvianet]

monit_debugger.vault_ctl.log

MONIT-DEBUG date
Thu Jun 21 19:21:00 UTC 2018
MONIT-DEBUG env
MONIT_PROCESS_PID=0
MONIT_HOST=localhost
MONIT_PROCESS_CPU_PERCENT=0
MONIT_DATE=Thu, 21 Jun 2018 19:21:00 +0000
MONIT_PROCESS_CHILDREN=0
MONIT_DESCRIPTION=Started
PATH=/bin:/usr/bin:/sbin:/usr/sbin
MONIT_PROCESS_MEMORY=0
MONIT_SERVICE=vault
PWD=/etc/sv/monit
MONIT_EVENT=Started
MONIT-DEBUG vault_ctl /var/vcap/jobs/vault/bin/vault_ctl start
monit_debugger.vault_ctl.log (END)

vault.log

$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/usr/sbin:/usr/bin:/sbin:/bin
skip empty key
skip empty key
skip empty key
skip empty key
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
wait_pid 4450 1 25 1 250
Killing /var/vcap/sys/run/vault/vault.pid: 4450 
.Stopped
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/usr/sbin:/usr/bin:/sbin:/bin
skip empty key
skip empty key
skip empty key
skip empty key

[Alexvianet]

i think /var/vcap/jobs/vault/data/unseal_keys include 3 kays and one empty line can be reason.

[jhunt]

It looks like you're running 1.0.1, which includes a test to skip empty keys. It prints skip empty key in the log, which I see in your output. Is it possible that it is treating all of the keys as empty? I see it in their four times after the $PATH ... lines that occur on every startup.

[Alexvianet]

vault 1.0.2* 2287f1c

and my /var/vcap/jobs/vault/data/unseal_keys

ixCAjtYxxxxxxxxxxxxxxxxxxxxxxxxxxqQCN/W7f
aEQ2tGxxxxxxxxxxxxxxxxxxxxxxxxlsDzu1yRab
hns/oswxxxxxxxxxxxxxxxxxxxxxxxxxxxxZqjsqR
                                                                     <--- empty line
~

vault.log

$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/usr/sbin:/usr/bin:/sbin:/bin
skip empty key
skip empty key
skip empty key
skip empty key
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
wait_pid 4400 1 25 1 250
Killing /var/vcap/sys/run/vault/vault.pid: 4400 
.Stopped
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
wait_pid 15482 1 25 1 250
Killing /var/vcap/sys/run/vault/vault.pid: 15482 
Stopped
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/bin:/usr/bin:/sbin:/usr/sbin
$PATH /var/vcap/packages/vault/bin:/var/vcap/packages/ttar/bin:/var/vcap/packages/node_exporter/bin:/var/vcap/packages/envconsul/bin:/var/vcap/packages/consul/bin:/var/vcap/packages/consul-template/bin:/usr/sbin:/usr/bin:/sbin:/bin
skip empty key
skip empty key
skip empty key
skip empty key

[Alexvianet]

OMG vault.skip_verify fix my problem, sorry for disturb.

[jhunt]

No worries, glad you figured it out!