search

hashicorp / consul-template

Template rendering, notifier, and supervisor for @HashiCorp Consul and Vault data.
https://www.hashicorp.com/
Mozilla Public License 2.0
4.76k stars 783 forks source link

Consul-template doesn't render with keyOrDefault if it gets a 'connection refused' #957

Closed LordBrain closed 7 years ago

LordBrain commented 7 years ago

Consul Template version

0.18.3

Configuration

template {
  source = "/opt/ctmpl/sensu/metrics_kafka_lag.json.ctmpl"
  destination = "/etc/sensu/conf.d/checks/metrics_kafka_lag.json"
  perms = 0644
  command = "/bin/bash -c 'chown root:sensu /etc/sensu/conf.d/checks/metrics_kafka_lag.json && systemctl restart sensu-client'"
}
{{$app := env "APPLICATION" -}}
{{$stack := env "STACK" -}}
{{$env := env "ENVIRONMENT" -}}

{
  "checks": {
    "metrics_kafka_lag": {
      "comment": "Check lag for all topics for this group.",
      "command": "metrics-kafka-topic.rb --scheme :::metric_prefix:::.kafka --group {{ keyOrDefault (printf "%s/%s/%s/input-kafka-group-id" $app $stack $env) "gcp-lab" }} --zookeeper {{ keyOrDefault (printf "%s/%s/%s/input-kafka-zk-connect" $app $stack $env) "elk-zookeeper.service.consul" }}:2181 2>/tmp/metrics_kafka_lag.stderr",
      "standalone": true,
      "type": "metric",
      "handlers": [
        "graphite"
      ],
      "interval": 30
    }
  }
}

Command

/usr/local/bin/consul-template -once -template "/opt/ctmpl/sensu/metrics_kafka_lag.json.ctmpl:/etc/sensu/conf.d/checks/metrics_kafka_lag.json:chmod 0644"

Debug output

2017/06/13 19:24:10.962782 [INFO] consul-template v0.18.3 (74151f8)

2017/06/13 19:24:10.962814 [INFO] (runner) creating new runner (dry: false, once: true)

2017/06/13 19:24:10.963315 [DEBUG] (runner) final config: {"Consul":{"Address":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":5,"Backoff":250000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"trace","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"chmod 0644","CommandTimeout":30000000000,"Contents":"","Destination":"/etc/sensu/conf.d/checks/metrics_kafka_lag.json","Exec":{"Command":"chmod 0644","Enabled":true,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/opt/ctmpl/sensu/metrics_kafka_lag.json.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":""}],"Vault":{"Address":"","Enabled":false,"RenewToken":true,"Retry":{"Attempts":5,"Backoff":250000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}}

2017/06/13 19:24:10.963353 [INFO] (runner) creating watcher

2017/06/13 19:24:10.963496 [INFO] (runner) starting

2017/06/13 19:24:10.963509 [DEBUG] (runner) running initial templates

2017/06/13 19:24:10.963522 [INFO] (runner) initiating run

2017/06/13 19:24:10.963537 [DEBUG] (runner) checking template 6772a9bf786ee06dc1c3997367815c02

2017/06/13 19:24:10.963873 [DEBUG] (runner) was not watching 2 dependencies

2017/06/13 19:24:10.963890 [DEBUG] (watcher) adding kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id)

2017/06/13 19:24:10.963895 [TRACE] (watcher) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting

2017/06/13 19:24:10.963898 [DEBUG] (watcher) adding kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect)

2017/06/13 19:24:10.963901 [TRACE] (watcher) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting

2017/06/13 19:24:10.963905 [DEBUG] (runner) diffing and updating dependencies

2017/06/13 19:24:10.963907 [DEBUG] (runner) watching 2 dependencies

2017/06/13 19:24:10.963935 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:10.963948 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:10.964165 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:10.964177 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:10.964500 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 1 after "250ms")

2017/06/13 19:24:10.964680 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 1 after "250ms")

2017/06/13 19:24:11.214717 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:11.214798 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:11.214944 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:11.215009 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:11.215318 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 2 after "500ms")

2017/06/13 19:24:11.215500 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 2 after "500ms")

2017/06/13 19:24:11.715487 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:11.715556 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:11.715592 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:11.715602 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:11.715846 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 3 after "1s")

2017/06/13 19:24:11.716519 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 3 after "1s")

2017/06/13 19:24:12.716022 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:12.716075 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:12.716346 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 4 after "2s")

2017/06/13 19:24:12.716648 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:12.716665 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:12.716818 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 4 after "2s")

2017/06/13 19:24:14.716539 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:14.716596 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:14.716913 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:14.716928 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:14.717047 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 5 after "4s")

2017/06/13 19:24:14.717112 [WARN] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (retry attempt 5 after "4s")

2017/06/13 19:24:18.717191 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id) starting fetch

2017/06/13 19:24:18.717231 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-group-id): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-group-id?stale=true&wait=1m0s

2017/06/13 19:24:18.717443 [TRACE] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect) starting fetch

2017/06/13 19:24:18.717456 [TRACE] kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): GET /v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=true&wait=1m0s

2017/06/13 19:24:18.717626 [ERR] (view) kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused (exceeded maximum retries)

2017/06/13 19:24:18.717652 [ERR] (runner) watcher reported error: kv.get(logging/offsetchecker/pe-ci/input-kafka-zk-connect): Get http://127.0.0.1:8500/v1/kv/logging/offsetchecker/pe-ci/input-kafka-zk-connect?stale=&wait=60000ms: dial tcp 127.0.0.1:8500: getsockopt: connection refused

Expected behavior

Consul-template would render the file with the default key.

Actual behavior

Consul template does not render the file. Which causes testing to fail.

Steps to reproduce

  1. Stop consul service
  2. Run consul-template -once ....

References

sethvargo commented 7 years ago

Hi @blee1170

Thank you for opening an issue. This is behaving as intended. The keyOrDefault function attempts to read a key in Consul. If no key exists at that path, the default value is rendered instead. However, a failure to connect to Consul or Vault is always an error. You can control retries globally or on a per-template basis using the retry stanza. Thanks! 😄

LordBrain commented 7 years ago

Hi @sethvargo

Our use case might be a little unique, but it might also be useful for other people. It would be nice to pass a flag to allow the default to render even if it can not talk to consul. In our case we want to validate that the files get placed when we create a RPM, but before its deployed to any environment, hence we are getting the connection refused.

Can we turn this into a feature request?

sethvargo commented 7 years ago

Hi @blee1170

Consul Template's architecture doesn't allow for this kind of behavior, and implementing such a feature would require a massive refactor of the codebase and internal data flow.

LordBrain commented 7 years ago

That doesn't mean it still cant be a feature request. Sometimes people will want to render a file using consul template using just environment variables. This will be a nice feature to have.