search

hashicorp / consul-template

Template rendering, notifier, and supervisor for @HashiCorp Consul and Vault data.
https://www.hashicorp.com/
Mozilla Public License 2.0
4.76k stars 782 forks source link

'secrets' query does not renders out.txt #1462

Closed Dentrax closed 3 years ago

Dentrax commented 3 years ago

Consul Template version

consul-template v0.25.2 (870905d) Vault v1.7.0 ('4e222b85c40a810b74400ee3c54449479e32bb9f+CHANGES')

Configuration

in.tpl

{{ range secrets "secret/" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/" }}
{{ with secret (printf "secret/%s" .) }}{{ range $k, $v := .Data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

Command

$ vault server -dev -dev-root-token-id=root -log-level=trace

P.S: I can not see any requests logs here after run consul-template CLI.

Debug output

https://paste.ubuntu.com/p/SqTPQGJm2W/

Expected behavior

It should render:

bar
foo
zip

Actual behavior

$ cat out.txt => EMPTY

Steps to reproduce

  1. $ vault server -dev -dev-root-token-id=root -log-level=trace
  2. $ vault kv put secret/foo bar=baz
  3. $ vault kv list secret/ 
    Keys
----
foo
qux
  4. $ consul-template -template "in.tpl:out.txt" -vault-renew-token=false -vault-token='root' -once => OK
  5. $ cat out.txt => EMPTY

Trace logs:

``` [INFO] consul-template v0.25.2 (870905d) [INFO] (runner) creating new runner (dry: false, once: true) [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"debug","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"out.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"in2.tpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":null,"Max":null},"Once":true,"BlockQueryWaitTime":60000000000} [INFO] (runner) creating watcher [INFO] (runner) starting [DEBUG] (runner) running initial templates [DEBUG] (runner) initiating run [DEBUG] (runner) checking template c23f0a4c34465b4756b6e34817ef0d6b [DEBUG] (runner) missing data for 1 dependencies [DEBUG] (runner) missing dependency: vault.list(secret) [DEBUG] (runner) add used dependency vault.list(secret) to missing since isLeader but do not have a watcher [DEBUG] (runner) was not watching 1 dependencies [DEBUG] (watcher) adding vault.list(secret) [DEBUG] (runner) diffing and updating dependencies [DEBUG] (runner) watching 1 dependencies [DEBUG] (runner) receiving dependency vault.list(secret) [DEBUG] (runner) initiating run [DEBUG] (runner) checking template c23f0a4c34465b4756b6e34817ef0d6b [DEBUG] (cli) receiving signal "urgent I/O condition" [DEBUG] (runner) rendering "in2.tpl" => "out.txt" [DEBUG] (runner) diffing and updating dependencies [DEBUG] (runner) vault.list(secret) is still needed [DEBUG] (runner) watching 1 dependencies [DEBUG] (runner) all templates rendered [INFO] (runner) once mode and all templates rendered [INFO] (runner) stopping [DEBUG] (runner) stopping watcher [DEBUG] (watcher) stopping all views ```

References

cc: @developer-guy

yilmazo commented 3 years ago

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}
eikenb commented 3 years ago

1468 is merged and fixes this.

rishianand06 commented 3 years ago

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

My Environment - consul-template v0.25.2 (c8372a0a) Vault v1.7.0-rc1 (9af08a1c5f0f855984a1fa56d236675d167f578e)

Put secrets in vault - vault kv put secret/tools/tool1/token/acc1 key1='vaule1' vault kv put secret/tools/tool1/token/acc1 key2='vaule1'

vault kv put secret/tools/tool1/token/acc2 key3='vaule2'

Vault cli command output - vault kv list secret/tools/tool1/token Keys

acc1 acc2

vault kv get secret/tools/tool1/token/acc1 ============= Data ============= Key Value

key1 value1 key2 value1

vault kv get secret/tools/tool1/token/acc2 ============= Data ============= Key Value

key3 value2

Consul templates - Now I need to write consul templates to read K/V pairs for each account (acc1, acc2,.....). Tried to do the same way, as you told.

in.tpl {{ range secrets "secret/tools/tool1/token/metadata" }} {{ with secret (printf "secret/tools/tool1/token/data/%s" .) }}{{ range $k, $v := .Data.data }} {{ $k }}: {{ $v }} {{ end }}{{ end }}{{ end }}

Run
consul-template -template "in.tpl:out.txt" -once

Actual behavior $ cat out.txt => EMPTY

Expected behavior It should render:

key1 value1 key2 value1 key3 value2

Please let me know, if I am missing something.

Thanks

eikenb commented 3 years ago

Hey @rishianand06,

Have you tried with a build from the master branch yet, this should be fixed there. It (the fix) will also be included in version 0.26.0, which I'll be releasing sometime in the next week or two.

If this was tried on the master branch and it didn't fix it for you please let me know. Thanks.

rishianand06 commented 3 years ago

@eikenb No I haven't tried that yet. Will try and let you know in case of any issue/s. Thanks :)