feature request: support for kubernetes api HTTP proxy so that tools like helm work with cloudflared

[cortopy]

I'm a new user of cloudflared and although setup has been quite bumpy, I have come to understand what it achieves and can do. And I love it!

The new support of kubectl is just what I needed to make the final jump but it still falls short of providing full support for kubernetes tooling.

In particular, I'm thinking of helm and helm-related tools. The setup recommended in the docs works by providing a tunnel via HTTPS_PROXY env variable. That's fine for kubectl since all commands consume the kubernetes api and there is no need to access the public internet.

However, as I understand it, the kubectl ecosystem expects HTTPS_PROXY to be just that (i.e.: a http proxy with access to the internet and capable of doing routing). It seems that the command

cloudflared access tcp --hostname cluster.site.com --url 127.0.0.1:1234

is not providing that, but a tcp/socks5 proxy that routes all traffic to the same endpoint.

By not following the assumption that HTTPS_PROXY is meant to be used for the http protocol, cloudflared makes helm to fail. I've tried following the same logic for helm,and so I may have:

alias helm="env HTTPS_PROXY=socks5://127.0.0.1:6444 helm"

This makes commands like helm list to fully work since there is only one routing to the api server. However commands like helm repo update or helm repo add fail because the HTTPS_PROXY is not routing the request to the public internet. Helm fails because for every https request it makes, it gets the same tls certificate back: that of the kubernetes api server.

I've managed to make helm work with a setup which is far more complex than I would like to.

With something like this:

alias kubectl="env HTTPS_PROXY=socks5://127.0.0.1:1234 kubectl"
alias helm="helm --kube-apiserver 127.0.0.1:6443"
kubectl proxy --port=6443

I get to achieve the minimum required:

• kubectl connects via argo tunnel
• kubectl forwards the api server to the local network
• helm does not use HTTPS_PROXY but instead knows that when making requests to the api, it should use localhost instead of the server address in kubeconfig

In an ideal world, I would guess that something like:

cloudflared access http --hostname cluster.site.com --url 127.0.0.1:1234

would solve the problem in terms of providing that extra layer of an http proxy, instead of tcp/socks5.

I don't know how complex this would be to implement, but it seems to me like it would just be the missing cherry on top

[michailbrynard]

I agree HTTP proxy support would be useful. Commands like kubectl exec also don't work with cloudflared, although it looks like this is being addressed in https://github.com/kubernetes/kubernetes/pull/84205.

Currently another workaround is to use Polipo to convert the SOCKS5 proxy into an HTTP proxy.

cloudflared access tcp --hostname cluster.site.com --url 127.0.0.1:1234
polipo socksParentProxy=127.0.0.1:1234
env HTTPS_PROXY=http://127.0.0.1:8123 kubectl exec -it ubuntu -- bash

[terinjokes]

When I tested previously, there was issues with some of kubectl's authentication scenarios, specifically client certificates, when using an HTTP proxy. Generally, SOCKS5 is easier to work with by virtue of being much more simple: you're not putting HTTP inside of another HTTP connection.

If you're running a kubectl from v1.19 or later, you can set proxy-url in your cluster configuration in kubeconfig, which should hopefully resolve your issues with Helm. (If you also use the patch from kubernetes/kubernetes#84205, as I do, then exec and port-forward would also work).

By not following the assumption that HTTPS_PROXY is meant to be used for the http protocol.

This environment variable tells applications how to proxy HTTPS traffic. While this has historically been an "http proxy", it is increasingly common to support SOCKS5 proxying as well. The Go standard library has long supported this, and it's been supported by curl (and likely anything linked to libcurl) for over a decade.

but a tcp/socks5 proxy that routes all traffic to the same endpoint.

This a limitation of cloudflared and not the SOCKS5 protocol. It'd like to see this revisited: ideally cloudflared accepts all traffic route to the correct tunnel when known based on hostname, and otherwise route the traffic locally. Without a refactor of how this works, a proposed "cloudflared access http" would likely have this same limitation.

Currently another workaround is to use Polipo to convert the SOCKS5 proxy into an HTTP proxy.

That's a nifty workaround! I think I've seen a few other projects try to tackle bridging those two. (As an aside, in a similar vain, I had some success with an experiment using socat to turn the SOCKS5 proxy into a tuntap interface, and assigned the IP of my Kubernetes server to the virtual interface. While fun, I my goal is to run less binaries to connect to Kubernetes, not more!)

In the end, I'm hoping to get https://github.com/kubernetes/kubernetes/pull/84205 (or similar) merged so you no longer need such a workaround!