hashicorp / consul-template

Template rendering, notifier, and supervisor for @HashiCorp Consul and Vault data.
https://www.hashicorp.com/
Mozilla Public License 2.0
4.76k stars 781 forks source link

tree function ignores keys ending with / #1712

Open phemmer opened 1 year ago

phemmer commented 1 year ago

Please note that the Consul Template issue tracker is reserved for bug reports and enhancements. For general usage questions, please use the Consul Community Portal or the Consul mailing list:

https://discuss.hashicorp.com/c/consul
https://groups.google.com/forum/#!forum/consul-tool

Please try to simplify the issue as much as possible and include all the details to replicate it. The shorter and simpler the bug is to reproduce the quicker it can be addressed. Thanks.

Consul Template version

e04ca78

Configuration

consul {
    address = "localhost:8500"
}

template {
    contents = "{{ range $kv := tree \"/\" }}{{ $kv.Key }}={{ $kv.Value }}\n{{ end }}"
    destination = "/tmp/out"
}
# consul kv put foo 1
# consul kv put foo/ 2
# consul kv put foo/bar 3

Command

./consul-template -once -config consul-template.conf -log-level trace

Debug output

2023-02-21T19:43:36.462-0500 [INFO] consul-template v0.30.0 ()
2023-02-21T19:43:36.462-0500 [INFO] (runner) creating new runner (dry: false, once: true)
2023-02-21T19:43:36.463-0500 [DEBUG] (watcher) adding vault.token
2023-02-21T19:43:36.463-0500 [TRACE] (watcher) vault.token starting
2023-02-21T19:43:36.463-0500 [TRACE] (view) vault.token starting fetch
2023-02-21T19:43:36.463-0500 [TRACE] vault.token: starting renewer
2023-02-21T19:43:36.463-0500 [DEBUG] (runner) final config: {"Consul":{"Address":"localhost:8500","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"trace","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{ range $kv := tree \"/\" }}{{ $kv.Key }}={{ $kv.Value }}\n{{ end }}","CreateDestDirs":true,"Destination":"/tmp/out","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":""}],"TemplateErrFatal":null,"Vault":{"Address":"","Enabled":false,"Namespace":"","RenewToken":true,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":null,"Max":null},"Once":true,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
2023-02-21T19:43:36.463-0500 [INFO] (runner) creating watcher
2023-02-21T19:43:36.463-0500 [INFO] (runner) starting
2023-02-21T19:43:36.463-0500 [DEBUG] (runner) running initial templates
2023-02-21T19:43:36.463-0500 [DEBUG] (runner) initiating run
2023-02-21T19:43:36.463-0500 [DEBUG] (runner) checking template ca09035a33eb5f87be732366ed54229c
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) missing data for 1 dependencies
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) missing dependency: kv.list(/)
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) add used dependency kv.list(/) to missing since isLeader but do not have a watcher
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) was not watching 1 dependencies
2023-02-21T19:43:36.464-0500 [DEBUG] (watcher) adding kv.list(/)
2023-02-21T19:43:36.464-0500 [TRACE] (watcher) kv.list(/) starting
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) diffing and updating dependencies
2023-02-21T19:43:36.464-0500 [DEBUG] (runner) watching 1 dependencies
2023-02-21T19:43:36.464-0500 [TRACE] (view) kv.list(/) starting fetch
2023-02-21T19:43:36.464-0500 [TRACE] kv.list(/): GET /v1/kv//?stale=true&wait=1m0s
2023-02-21T19:43:36.465-0500 [TRACE] kv.list(/): returned 3 pairs
2023-02-21T19:43:36.465-0500 [TRACE] (view) kv.list(/) marking successful data response
2023-02-21T19:43:36.465-0500 [TRACE] (view) kv.list(/) successful contact, resetting retries
2023-02-21T19:43:36.465-0500 [TRACE] (view) kv.list(/) received data
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) receiving dependency kv.list(/)
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) initiating run
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) checking template ca09035a33eb5f87be732366ed54229c
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) rendering "(dynamic)" => "/tmp/out"
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) diffing and updating dependencies
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) kv.list(/) is still needed
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) watching 1 dependencies
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) all templates rendered
2023-02-21T19:43:36.465-0500 [INFO] (runner) once mode and all templates rendered
2023-02-21T19:43:36.465-0500 [INFO] (runner) stopping
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) stopping watcher
2023-02-21T19:43:36.465-0500 [DEBUG] (watcher) stopping all views
2023-02-21T19:43:36.465-0500 [TRACE] (watcher) stopping kv.list(/)
2023-02-21T19:43:36.465-0500 [DEBUG] (runner) stopping vault token watcher
2023-02-21T19:43:36.465-0500 [DEBUG] (watcher) stopping all views
2023-02-21T19:43:36.465-0500 [TRACE] (watcher) stopping vault.token

Expected behavior

/tmp/out should contain:

foo=1
foo/=2
foo/bar=3

Actual behavior

/tmp/out contains:

foo=1
foo/bar=3

Steps to reproduce

Provided above

References

phemmer commented 1 year ago

This is apparently deliberate behavior, but I have no idea why it would be desired to skip keys.

https://github.com/hashicorp/consul-template/blob/e04ca78a334bb5f129cac46326458404d8a968c3/template/funcs.go#L644-L645

This fixes it: https://github.com/hashicorp/consul-template/commit/b9ac2481ebb0b3b2f64e44ff6eee7e80dbdb33cd