Open phemmer opened 3 years ago
phemmer
commented
3 years ago After a bunch of experimentation, I found an evil workaround:
{{ with key "mykey" }}
{{ secret (print "secret/mysecret?version=-" . }}
{{ end }}This passes mykey as a negative version number. Vault seems to ignore negative version numbers and retrieves the latest value. However consul-template doesn't know this and uses it as the cache key.
eikenb
commented
3 years ago Hey @phemmer, thanks for taking the time to file this ticket.
The behavior is as intended as each value being monitored separately and triggers the template when it gets a new value, using the new value for it and cached values for all other entries. That is when a the monitoring process finds an update, it updates the value in the cache and triggers the template.
So what is needed here is a way to force trigger a value fetch when another one is triggered. Basically an indirect dependency. This dependency relationship needs to be specified and stored somehow, either in the template or in the config. Currently leaning toward in the template as that's where other dependencies are expressed.
otto-dev
commented
1 year ago If you change a secret using vault kv put, the render output stays the same. Even if you manually re-render the template, it uses the cached version. This can't be the intended behavior? That's what I observe in my Nomad template blocks which are written to file.
Changing a secret that's used in a template should trigger the template. It doesn't.
Rendering a template afresh should use the latest value from the vault. Also not the case.
That means you can never change a secret used in consul-template.
otto-dev
commented
1 year ago The Renew API tries to use most of the time the secret is good, renewing at around 90% of the lease time (as set by Vault).
My guess would be that the bug is because the Renewer thinks the old secret still has life left and there is no need to refresh it (even if changed in vault). Hence you are stuck on the old secret for X amount of time. I don't know if X is finite.
Wnthr
commented
1 year ago If you change a secret using
vault kv put, the render output stays the same. Even if you manually re-render the template, it uses the cached version. This can't be the intended behavior? That's what I observe in myNomadtemplateblocks which are written to file.
So what it sounds like you are saying is that in a template with 100 different items, everytime one of them changes, all 100 should be refreshed? That is not clear to me to be the intended result at all in a caching system, but maybe I am missing something. The original response seems to indicate that all the values are indeed cached individually, and that not getting all the cached values is indeed intended behavior, and is what I would expect from a cache.
The functionality provided is not a cache of grouped values that go together on the template level, where all values would be updated on the individual secret level, and the process on invalidating one secret and triggering the template engine to put whatever is in the cache in our respective files is wholly separate from figuring out that a value changes in vault, no?
Changing a secret that's used in a template should trigger the template. It doesn't.
"Trigger the template" should indeed happen, but the triggering of the template should read the cached values (of which 99 are the same, and the one that actually changed is now populating that cache, from which said template is triggering? It seems as it does, as soon as the change detector realizes a value has changed, does it not?
Rendering a template afresh should use the latest value from the vault. Also not the case.
I would like to be able to have a brute force trigger available to invalidate the cache and force consul-template to re-read all secrets, but that seems to be easiest done by sending a SIGHUP to the consul-template. We might want that as an option inside of a template, but that seems like a different issue than what should be the normal behavior. It seems to me that as long as consul-template thinks a value in the cache is valid, it should use that instead of querying the original source?
Of course, that could trigger a discussion about whether the algorithm to figure out when a value is updated needs looking at, but that seems a wholly different discussion from whether the templates render correctly, doesn't it? If there is data on this not working as intended, I would be interested to see it.
That means you can never change a secret used in
consul-template.
That doesn't seem to follow, though. If you again, have these 100 values in the template that change, going through those secrets and checking for updates will be done on a schedule that defaults to "once every five minutes", my tests using two variables seem to indicate that I have the new updated versions of those secrets in the template output file within 5 minutes of changing them, and indeed, when setting the value of default_lease_duration to 30s instead, it does have all my new values in the file within those 30s.
What is seems to me, is that caching on the secret level is (and should be) per individual secret, not per template, and that renewal of those values happens (and should happen) within the constraints of the renewal system as configured. If I change my values, I will have those updates reflected in the output file for a given template within default_lease_duration.
If I am missing some erroneous behavior that you have experienced (I have not done any testing on the consul or nomad kv engines, for instance), please give some more specific information about what the conditions of that error is, so we can take a look and figure out what is going on. In the meantime, it seems to me after some testing that I will have to agree with @eikenb:
The behavior is as intended as each value being monitored separately and triggers the template when it gets a new value, using the new value for it and cached values for all other entries. That is when a the monitoring process finds an update, it updates the value in the cache and triggers the template.
Consul Template version
consul-template v0.25.1 (b11fa800)
Configuration
Consul
mykey:Vault
mysecret:Command
Debug output
Expected behavior
Secret value should have updated
Actual behavior
Secret value remained unchanged
Steps to reproduce
mysecretand set to{"foo":"bar2"}.mykeyto1234.Additional info
I'm assuming there's some caching going on here. While the documentation does state:
...it does not make it clear that the secret will not reload even if something else in the template triggers a render. If this is deliberate behavior, can we please get a mechanism for busting the cache.