consul-template 504 from outside kuberentes cluster and consul in cluster

[wadeholler]

Please note that the Consul Template issue tracker is reserved for bug reports and enhancements. For general usage questions, please use the Consul mailing list:

https://groups.google.com/forum/#!forum/consul-tool

Consul Template version

Run consul-template -v to show the version. If you are not running the latest version, please upgrade before submitting an issue.

consul-template v0.20.0 (b709612c)

Configuration

# Copy-paste your Consul Template configuration file here
consul {
    address = "obsfuscated.domain"
    auth {
      enabled = false
    }
    retry {
        enabled = false
        attempts = 12
        backoff = "250ms"
    }
    ssl {
      enabled = false
    }
}
reload_signal = "SIGHUP"
kill_signal = "SIGINT"
max_stale = "10m"
log_level = "warn"
wait {
    min = "5s"
    max = "10s"
}
template {
     source = "/home/centos/projects/learn-consul-template/test.tpl"
     destination = "/home/centos/projects/learn-consul-template/output"
}

# Copy-paste your Consul Template template here
{{ key "generic/influxdbaddr" }}

Command

# Place your Consul Template command here
consul-template  -config config

Debug output

Provide a link to a GitHub Gist containing the complete debug output by running with -log-level=trace.

2019/04/30 15:53:58.248009 [ERR] (view) kv.block(generic/influxdbaddr): Unexpected response code: 504 (exceeded maximum retries)
2019/04/30 15:53:58.248042 [ERR] (runner) watcher reported error: kv.block(generic/influxdbaddr): Unexpected response code: 504
2019/04/30 15:53:58.248061 [ERR] (cli) kv.block(generic/influxdbaddr): Unexpected response code: 504

Expected behavior

What should have happened?

I would like consul-template to work (communicating to consul) through kubernetes ingress which is behind aws ELB

Actual behavior

What actually happened?

504 message consistent with other cases of consul being behind a proxy

Steps to reproduce

1. run consul-template through kubernetes ingress on aws

References

Are there any other GitHub issues (open or closed) that should be linked here? For example:

• GH-1234
• ...

[eikenb]

Thanks for the report @wadeholler.

As I don't have ready access to a EKS cluster to test this I was wondering if there were ways to reproduce this locally. Using minikube or similar? Or you say "504 message consistent with other cases of consul being behind a proxy"... does that mean you can reproduce this using a different, perhaps local, proxy setup?

Thanks.

[wadeholler]

I don’t have any “local” clusters that I use. Only ones on and off Orem with full Ingress etc. test local with docker-dind perhaps ?(I can’t remember if it supports running a real ingress. )

[eikenb]

I'm currently looking into another similar issue, was looking this over again and had a couple thoughts.

First, have you seen this.. https://stackoverflow.com/questions/45711805/consul-template-unexpected-response-code-504 That describes setting the timeouts higher than the keepalive.

I'm also curious why you disabled retries. That would have been another way the system could have dealt with the disconnections.

I am sorry about the lack of progress on this ticket but until I can reproduce it or figure out the problem in some other way that doesn't involve an ELB, I'm at a bit of a standstill.