0.16 breaks simple key lookups

[jgerry]

Consul Template version

consul-template v0.16.0

Consul version

0.5.2

Configuration

consul = "<redacted>:8500"
retry  = "10s"
token  = "<redacted>"
max_stale = "10m"
log_level = "debug"

wait = "5s:10s"

auth {
  enabled  = false
  username = "test"
  password = "test"
}

template {
  source          = "/root/test.ctemp"
  destination     = "/root/test.out"
  command_timeout = "60s"
  perms           = 0644
  backup          = false
}

[monitor:///root/production.log]
index = {{ key "velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX" }}
sourcetype = ruby_on_rails
host = gapi-{{env "ENV"}}-{{env "IP"}}
disabled = 0

Command

ENV=test1 IP=1.2.3.5 ./consul-template-1.6 -config="consul-template-2.conf"

Debug output

https://gist.github.com/jgerry/5a86513785e9333592bd05ed7fb1be9b

Expected behavior

template should have generated the config file normally. this works fine with 0.14 and 0.15.

Actual behavior

[WARN] ("key(velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX)") Consul returned no data (does the path exist?) is thrown. the key does exist, this config works fine for 0.14 and 0.15

Steps to reproduce

1. run command above using configs and template files above
2. error occurs immediately

[sethvargo]

Hi @jgerry

That is just a warning. You can see in these lines of your debug log that the template is being rendered.

I'm not seeing any errors in your output.

[jgerry]

Let me clarify: the template is being generated, but the value of key SPLUNK_INDEX is rendering as blank. in 0.14 and 0.15 it is being generated properly with the value of SPLUNK_INDEX.

[sethvargo]

Hi @jgerry

Can you confirm that there is a value for Consul at that particular path? Maybe look in the webui or just use curl:

$ curl http://localhost:8500/v1/kv/velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX

[jgerry]

The value definitely exists:

$ consul-cli kv read "velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX" --consul="<redacted>:8500"
man_gapi_preprod

[sethvargo]

Hi @jgerry

And these are the same Consul cluster and IP? Is there a reason you are not querying localhost:8500 directly? The warning means that the API is not returning a value at that key. That warning has always existed.

The only real change between 0.15.0 and 0.16.0 was the removal of the enforcement of the trailing slash. Can you show me the output of the following please?

curl http://localhost:8500/v1/kv/velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX

[jgerry]

Same Consul cluster / IP, yes. Not using localhost as I'm not on the cluster itself, querying from another host, using an ACL.

curl http://<redacted>:8500/v1/kv/velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX\?token\=<redacted>
[{"CreateIndex":2601214,"ModifyIndex":2601242,"LockIndex":0,"Key":"velocity/configs/gateway_api/test1/v1/SPLUNK_INDEX","Flags":0,"Value":"bWFuX2dhcGlfcHJlcHJvZA=="}]%

I have actually put the binaries for 0.14, 0.15, and 0.16 together in the same directory I'm testing from. 0.14 and 0.15 work fine, the exact same command line and configs for 0.16 throw this warning and fail to render the value. 0.14 and 0.15 throw no warnings.

[jgerry]

Also note our Consul version is 0.5.2 - the team is working on upgrading that to current 0.7.0 but no ETC on that work.

[sethvargo]

Hi @jgerry

I've tried to reproduce this issue and I've been unable to. Given that 0.16.0 has been out for quite some time and this is the first reported issue, I'm inclined to believe there is something differing in your setup from mine.

Could you share any additional information about your Consul setup? In particular:

1. Does this reproduce if you don't use an ACL (try a different, un-ACLed key please)?
2. Does this reproduce if you query from inside your data center using the local Consul agent?
3. Are there any other setup differences you may have? (firewalls, running behind a load balancer, etc)?

[jgerry]

@sethvargo I'm with you, the whole thing seems super odd. I'll go through your steps and see what I can come up with. It'll probably be a day or two to find the time. And thanks for looking into it.

[sethvargo]

No problem - I'm just baffled. Sorry I don't have a good answer 😦

[sethvargo]

Hi @jgerry

I've tried a few times, and I'm unable to reproduce this. Without additional information or user-reports, I'm afraid this might be an environmental issue related to your setup. I'm going to close this issue out, but if you stumble across any additional information, please do let me know. I would be happy to jump back into this at any time. Thanks and sorry!