Warn on missing dependencies or data blocking template rendering

rwiggins commented 3 years ago

TL;DR: It would be cool if templates that are waiting on blocking queries had a warning printed after some amount of time indicating what they're waiting on. In our case, a missing Consul key blocked the rendering of a (complicated) template, which resulted in some head scratching until we hunted down the missing key.

Consul Template version

$ consul-template -v
 v0.26.0 (3b7f233a)

Configuration

template {
  source        = "/tmp/ct-test/test.ctmpl"
  destination   = "/tmp/ct-test/out.txt"
}

{{ key "missingkey" }}

(Any key not in the K/V store will suffice, missingkey is just an example)

Command

$ consul agent -dev # in another shell, easy-to-produce empty K/V store
$ consul-template -config /tmp/ct-test/config.hcl -log-level=debug

Debug output

On startup:

https://gist.github.com/rwiggins/29d1d7da822e16bbf482613a6f9d2c5d

then once the key is added with $ consul kv put testkey 1:

https://gist.github.com/rwiggins/29d1d7da822e16bbf482613a6f9d2c5d#gistcomment-3806914

Actual behavior

Rendering of the template is blocked until the key is added to Consul, specifically without any log messages indicating that (except at debug level).

In other words, Consul Template is executed, and it appears "hung" to the user — no logs, no warnings, just no template, until an investigation is made. In a simple template like the above, it's not really a mystery... however, this issue took a bit of hunting recently in one of our test builds, which has a substantially more complicated template.

Expected behavior

I understand that the blocking {{ key }} query is both desired and documented. However, I guess my hope would be a warning message after some time, under the assumption that it's atypical for blocking queries to take more than e.g. 5 seconds. Maybe if a template is blocked because of missing dependencies for N seconds, produce a warning message that enumerates the missing data? For example,

[WARN] (runner) template "/tmp/ct-test/test.ctmpl" rendering waiting on missing dependencies: kv.block(testkey), kv.block(testkey2)

I am not entirely sure how long to wait before warning. Maybe a second or two or five?

The example above only shows missing Consul keys, but I'm sure the same could apply to all sorts of other data sources. (See 2nd referenced issue below, for example.)

This type of warning message would have turned a recent puzzled investigation into an immediate resolution, at least in our use of Consul Template.

References

adjieguntoro-tokped commented 1 year ago

any update for this?

moonbaseDelta commented 1 year ago

the problem still a problem

eikenb commented 1 year ago

Hello @rwiggins, @adjieguntoro-tokped, @moonbaseDelta. Sorry for the delay looking more into this.

So the general ask here is to have it emit an log message if the key hasn't been fetched in "a while"... That the use cases aren't covered by one of the current ways to handle keys; keyExists, keyOrDefault and the error_on_missing_key template config option?

I'm looking to see if there is a quick way to address this ask but haven't found it yet and wanted to be sure everyone knew about the alternatives.

rwiggins commented 1 year ago

Hi! Thanks for taking a look.

I have since changed employers, but our templates were pretty large, basically a way to deploy flags/values fleet-wide at runtime for apps, e.g.:

my_first_key: {{ key "a/b/c" }}
my_second_key: {{ key "c/b/a" }}
# ... etc

There were templates with, say, O(100) keys in them, to set context.

{{ keyOrDefault }} might have been workable, although in our case there were keys you definitely didn't want falling back to some kind of generic default (at least not without special handling) - for example, I remember we ended up deploying a denylist through Consul Template, and that's the kind of thing you don't want to fail open. Not to say it can't work - code can be written - but again, more a question of ergonomics. The multi-phase execution model makes handling a case like that a little harder, too; you can't just write code that says if len(denyList) == 0 { panic("missing denylist"); }. For a case like this, I think {{ keyExists }} would have been roughly as useful as {{ keyOrDefault }}, i.e.: workable theoretically but an irritation.

As for error_on_missing_key: correct me if I'm wrong, but isn't that for a missing key in a map/struct, rather than a missing Consul {{ key }}? e.g., I think that makes a situation like this:

{{ with secret "a/b/c" }}
{{ .Data.foobar_key_that_isnt_real }}
{{ end }}

produce an error.

In retrospect, I am not entirely sure we were using Consul Template in the best way, but it was pretty easy to set up and worked surprisingly well. I think the only real hard-to-debug thing we ever had was this bug. At first glance, it would look like the template "just isn't rendering" with zero log messages.

rwiggins commented 1 year ago

... that said, I just happened to see #1637, which would have worked just as well for us! (and may be easier to implement 😄 )

eikenb commented 1 year ago

@adjieguntoro-tokped, @moonbaseDelta .. would #1637 also address your needs? If we could get a consensus I could just close this one out in favor of it.

hashicorp / consul-template