Vault Agent doesn't honor SIGHUP Signal

[john-mensel-peoplefluent]

Describe the bug Hi friends! We've run into an interesting cert renewal problem with the Vault Agent.

We are using TLS certs+client cert in the Vault Agent to connect to our Vault Server. We have created templates that pull updated certs for the Vault Agent from the Vault server so that the Vault Agent can rotate its TLS certificates. Expectation is that if we send a SIGHUP signal to the Vault Agent, it will reload the TLS certificates that we have generated for it via Templates.

TLS certificate renewal works as expected and writes valid TLS certs to the right locations, but Vault Agent doesn't respond to the SIGHUP signal, and does not reload the certificates into its running config. If we restart the Vault Agent daemon, it loads the new certificates and works as expected.

To Reproduce Steps to reproduce the behavior:

1. Start Vault agent with functioning TLS certificates/key with a TTL of 5 minutes. Everything works great.
2. Vault Agent renews Certs via templates (see vault.hcl below).
3. Check the cert that is in use via curl -v https://127.0.0.1:8200/v1/sys/init. Observe that the original certificate is still running (as expected.)
4. Send SIGHUP signal to vault agent via cli: kill -1 $vaultpid or pkill -HUP vault. kill command returns 0, indicating success.
5. Check the cert that is in use via curl -v https://127.0.0.1:8200/v1/sys/init and find that it has not changed.
6. Observe the cert that the Template functions have written to disk when the daemon restarted, and see that it has changed via openssl x509 -in /usr/local/vault/client.crt -noout -text. Compare hashes of key that Templates have written to disk and verify that they match the client.crt as expected.
7. Check the output of journalctl -u vault and find that there is no logged reload of the agent.
8. Restart Vault Agent, and observe that the new certificates are in use via curl -v https://127.0.0.1:8200/v1/sys/init
9. If we stop periodically restarting the Vault Agent, the certs expire and the agent can no longer talk to the vault server.

Expected behavior That the templated certificates and key will be loaded and used by the Vault Agent after the Vault Agent service is sent a SIGHUP.

Environment:

• Vault Server Version (retrieve with vault status): 1.3.0
• Vault CLI Version (retrieve with vault version): Vault v1.3.1
• Server Operating System/Architecture: Ubuntu 16.04 LTS, Consul backend.

 % vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.3.0
Cluster Name    vault-cluster-7545xxxx
Cluster ID      <hidden>
HA Enabled      true
HA Cluster      https://10.0.x.x:8201
HA Mode         active

Vault server configuration file(s):

pid_file = "/usr/local/vault/vault.pid"
exit_after_auth = false

vault {
        address = "https://vault.example.com"
        ca_cert = "/usr/local/vault/server.crt"
        client_cert = "/usr/local/vault/client.crt"
        client_key  = "/usr/local/vault/client.key"
        tls_server_name =  "vault.example.com"
}

auto_auth {
        method "approle" {
        mount_path = "auth/approle"
            config = {
                role_id_file_path = "/usr/local/vault/role-id"
                secret_id_file_path = "/usr/local/vault/secret-id"
                remove_secret_id_file_after_reading = "false"
           }
        }
        sink "file" {
                config = {
                        path = "/var/vault/tokensink"
                }
        }

}

cache {
        use_auto_auth_token = true
}

listener "tcp" {
         address = "127.0.0.1:8200"
         tls_disable = false
         tls_cert_file = "/usr/local/vault/combo.crt"
         tls_key_file  = "/usr/local/vault/client.key"
         tls_client_ca_file = "/usr/local/vault/server.crt"
}

template {
  source      = "/usr/local/vault/server.crt.tpl"
  destination = "/usr/local/vault/server.crt"
}

template {
  source      = "/usr/local/vault/client.crt.tpl"
  destination = "/usr/local/vault/client.crt"
}

template {
  source      = "/usr/local/vault/client.key.tpl"
  destination = "/usr/local/vault/client.key"
}

template {
  source      = "/usr/local/vault/combo.crt.tpl"
  destination = "/usr/local/vault/combo.crt"
}

Additional context Final Config for the runner seems to indicate that a ReloadSignal is not present:

Jan 22 15:30:01 somehostname vault[16544]: 2020/01/22 15:30:01.934805 [DEBUG] (runner) final config: {"Consul":{"Address":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/usr/local/vault/server.crt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/usr/local/vault/server.crt.tpl","Wait":{"Enabled":true,"Min":10000000000,"Max":40000000000},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/usr/local/vault/client.crt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/usr/local/vault/client.crt.tpl","Wait":{"Enabled":true,"Min":10000000000,"Max":40000000000},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath"
Jan 22 15:30:01 ip-<redacted> vault[16544]: :""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/usr/local/vault/client.key","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/usr/local/vault/client.key.tpl","Wait":{"Enabled":true,"Min":10000000000,"Max":40000000000},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/usr/local/vault/combo.crt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/usr/local/vault/combo.crt.tpl","Wait":{"Enabled":true,"Min":10000000000,"Max":40000000000},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""}],"Vault":{"Address":"https://vault.example.com","Enabled":true,"Grace":15000000000,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"/usr/local/vault/server.crt","CaPath":"","Cert":"/usr/local/vault/client.crt","Enabled":true,"Key":"/usr/local/vault/client.key","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false}

[john-mensel-peoplefluent]

Hi all! I'm curious to know if this is an expected behaviour, in which case I'll just continue with the workaround of using a cronjob to periodically restart the agent and add this as a feature request.

[drawks]

Yes, this does seem like a bug. I would expect that a hup signal would reload certs for both auto-auth as well as certs for TLS listeners. It is curious that this issue has both a bug and feature-request tag.

[pbar1]

Ah, looks like there's a TODO for this in the Agent's code

We're injecting the Vault Agent with expiring certs in the similar way (hashicorp/vault-k8s#213), so getting them SIGHUP'd would be great to avoid having to restart the whole pod.

[MagicRB]

I'm one upping this, I want to use this for auto-renewing certificate authenticated vault agent

[dekimsey]

Would addressing this also perhaps include template configuration reloading?

[quinndiggitypolymath]

Would be nice if vault agent could be used for anything; not particularly useful if it has to be restarted every day