Issue Renewing Token

[Emerson]

Howdy,

I hope I'm just doing something stupid here, but I can't seem to get consul-template renewing my vault token's correctly. Here is a quick breakdown:

1. I've used the AppRole auth to fetch a token manually from my vault instance
2. I set this token in my consul-template config
3. The token is valid and the template gets written correctly, pulling secrets out of vault
4. The token gets renewed. I can see the lease duration going down, and then being renewed by consul-template.
5. Consul-template does not appear to acknowledge the renewal and starts backing off the renew requests, which eventually causes my token to expire.

Consul Template version

v0.19.4

Configuration

{
  "vault": {
    "address": "http://10.x.x.x:8200",
    "grace": "1m",
    "token": "2fc272c1-a2f1-1dda-cb11-9b30ffb6bf8f",
    "renew_token": true,
    "ssl": {
      "enabled": false
    }
  },
  "log_level": "debug",
  "consul": [{
    "address": "127.0.0.1:8500"
  }],
  "template": [
    {
      "source": "/opt/consul-template/templates/env.tmpl",
      "destination": "/home/ubuntu/env"
    }
  ]
}

Debug output

[TRACE] vault.token: starting renewer
[TRACE] vault.token: successfully renewed
[WARN] vault.token: renewer returned (maybe the lease expired)
[TRACE] vault.token: token is not renewable, sleeping for 1m0s

Expected behavior

The vault token should be renewed periodically.

Actual behavior

The token is renewed, but consul-template does not appear to recognize the renewal and starts to back off renewing the token until it eventually just expires.

I'm monitoring the outbound HTTP traffic and can see that consul-template makes a successful renew call:

010.000.012.231.42958-010.000.011.143.08200: PUT /v1/auth/token/renew-self HTTP/1.1
Host: 10.0.11.143:8200
User-Agent: Go-http-client/1.1
Content-Length: 16
X-Vault-Token: 2fc272c1-a2f1-1dda-cb11-9b30ffb6bf8f
Accept-Encoding: gzip

{"increment":0}

[rberlind]

Hi @Emerson: What is the TTL on your token? Note that the consul-template docs on grace say that grace should be less than 1/3 of the token TTL.

It seems a bit unusual that the increment you saw is 0. I did some experiments with the Vault API and noticed that explicitly setting the increment to "0h" results in a TTL remaining on the token of 1 hour. But setting 0 without quotes seems to leave the current TTL unchanged. So, it is possible that consul-template is affectively renewing the lease without actually extending it. And it does appear that the consul-template code does not actually pass in an increment into the RenewerInput structure in https://github.com/hashicorp/vault/blob/master/api/renewer.go.

I work at HashiCorp and will consult with the Vault team and see if they agree with my analysis that consul-template should be passing an increment to Vault.

[rarguelloF]

Having same issue here, @rberlind do you know any workaround for this?

[rberlind]

I don't, but just sent an email to the Vault team to get their thoughts on the impact of not setting increment when renewing a token. If they agree with my finding that the TTL of the token is not extended, then the solution will presumably involve a code change to consul template to actually set an increment, ideally the original lease duration of the token.

Note that I modified my comment above about what happens when the increment is set to "0h" since I got different results depending on how my token had been generated and how the auth methods were tuned.

[rarguelloF]

I think I figured out what the problem is (at least in my case).

My setup was as follows:

• My refresh_interval parameter (which refers to the lease_duration parameter, see https://www.vaultproject.io/docs/secrets/kv/kv-v1.html#ttls) for the secrets I was reading from consul-template was set to 768h.
• My TTL was set to 1h (using the period parameter from the ec2 roles auth method).
• My grace period was set to 5m, which as the docs says, it's set to less or equal than 1/3 of the TTL.

So in this scenario, consul-template wouldn't try to renew the token until after 1h, when the token TTL would be indeed expired and could not be renewed anymore.

So, when I tweaked the values a little bit, for example, with secrets refresh_interval set to 5m, token period (TTL) to 10m and consul-template's grace parameter to 2m, consul-template is renewing the token before being expired and everything is working fine.

I think this issue is a documentation problem that could be fixed by clarifying consul-template's token renewal behaviour a little bit more.

EDIT: Regarding the increment parameter issue:

I played with the vault api's renew-self endpoint and (maybe only for the aws ec2 auth method, not sure), the increment parameter is ignored and the new TTL is always set again to the value of the token's period parameter.

[rberlind]

Glad you solved your problem @rarguelloF. I think there are a lot of parameters that can affect how the increment parameter is processed including the type of token, the current TTL, the max TTL, etc. I hope to do more testing in the next week.

[jefferai]

Note that Consul-Template may be out of date w.r.t. how it renews Vault tokens if it's allowing setting cutom grace; the current renewal logic in Vault's API client doesn't use user-provided grace periods at all. C-T may need to pull in dep updates.

[eikenb]

So short term this is a doc issue, but long(er) term it is still maybe a doc issue but also an issue with the vendored Vault needing an update. I'm still reviewing tickets, but I expect an update to the vendored dependencies will happen in the next (working on now) version.

[eikenb]

Now that the dependencies have been updated (part of the move to go modules) and Grace having been deprecated... is this still an issue at all? My reading of this is that the issue was due to a bad Grace setting and that is no longer possible. The documentation issue will be taken care of by #1266, as that covers removing the Grace docs completely.