Conditional proxy support

[phemmer]

Our environment requires the use of a proxy for outbound internet access. Thus we need proxy support for alert destinations like PagerDuty. Go does support the http_proxy and https_proxy environment variables, however these appear to control ALL http calls, which is not what we want as we do not want to use the proxy server for access to influxdb.

So we need some way of controlling what uses the proxy. 2 solutions I can think of:

1) Add a parameter to alert handlers which might need a proxy. E.G. alert().pagerDuty.proxy('http://foo:bar@proxyhost:3128').

or

[pagerduty]
    enabled = true
    proxy = "http://foo:bar@proxyhost:3128"

2) Add a per-site proxy list to the config. E.G.

[proxy]
    events.pagerduty.com = "http://foo:bar@proxyhost:3128"

^if toml is even capable of such a config

[nathanielc]

@phemmer Would it make sense to do something like?

[http_proxy]
    url = "http://foo:bar@proxyhost:3128"

[pagerduty]
    enabled = true
    use-proxy = true

That way you only have to configure the proxy once but then can use it for whichever services need it.

[phemmer]

seems acceptable to me.

[blaketmiller]

What's the status of this? Trying to get my TICK stack to production and I'm behind HTTP proxies as well, so this is a blocker for me.

Conditional proxying based on raising flags for each plugin seems...unwieldy, but how about getting just a basic proxy ability going for the entire service? Looks like this should be pretty easy to add in Go: https://golang.org/pkg/net/http/#ProxyFromEnvironment

[blaketmiller]

For anyone else waiting on this to get resolved, I made a quick workaround by tunneling through a relay: http://btmiller.com/2017/02/20/send-kapacitor-alerts-to-slack-through-a-proxy.html (only for Slack).

[phemmer]

We take a somewhat similar approach by using socat, and adding an /etc/hosts record for events.pagerduty.com to redirect to the local socat.

[desa]

Re-opened as per discussion https://github.com/influxdata/kapacitor/pull/1238#issuecomment-299004910

[desa]

@phemmer Just trying to gather a bit more information.

Our environment requires the use of a proxy for outbound internet access. Thus we need proxy support for alert destinations like PagerDuty. Go does support the http_proxy and https_proxy environment variables, however these appear to control ALL http calls, which is not what we want as we do not want to use the proxy server for access to influxdb.

Would it be sufficient for us to allow proxying for everything except InfluxDB? Rather than having conditional proxy support for each external service?

[phemmer]

In my use case that would be sufficient. The only thing that jumps to mind with that sort of solution is if there are other services that people might use that are hosted within their own network, and which would also need to exempt from the proxy (E.G. kubernetes).

[desa]

Good point about Kubernetes.

As an aside, #1238 may be sufficient for your use case. Go's ProxyFromEnvironment supports an environment variable NO_PROXY which will skip the proxy for any requests to that host.

[phemmer]

Ah, ok, NO_PROXY may indeed work. Wasn't familiar with this variable, and the documentation on ProxyFromEnvironment doesn't offer much (any) insight on how it works. But it seems like I should be able to do NO_PROXY=localhost (since in my case the InfluxDB server lives on localhost).

[desa]

I wasn't familiar with it until just recently myself. I think your point about having a larger strategy for interacting with internal vs external services still makes sense though.

[jcmartins]

cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)

rpm -qa |grep kapacitor kapacitor-1.4.1-1.x86_64

[root@kapacitor]# env |grep proxy http_proxy=http://proxy.intranet:80 https_proxy=http://proxy.intranet:80 no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com HTTPS_PROXY=http://proxy.intranet:80 HTTP_PROXY=http://proxy.intranet:80 NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com

My kapacitor.log ts=2018-04-24T14:46:27.800Z lvl=error msg="failed to send event" service=slack task=chronograf-v1-5c6eb4ed-678c-46d9-9407-6171b39892d6 err="Post https://hooks.slack.com/services/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: dial tcp 54.239.152.5:443: getsockopt: connection timed out" ts=2018-04-24T14:48:03.326Z lvl=error msg="error while sending usage report on startup" service=reporting err="Post https://usage.influxdata.com/api/v1/usage/kapacitor: dial tcp 104.131.151.204:443: i/o timeout"

[Namita-S]

Hi @nathanielc, What's the status of this? Trying to get my Chronograf to push slack alerts but it needs proxy to reach the Webhook url.

[bmgante]

Hi, i am facing this same issue. HTTP post to microsoft teams using proxy gets timeout even with env variables properly defined. Any idea?

[bmgante]

It seems to work defining the env vars directly on /etc/default/kapacitor