search

kubernetes / kubectl

Issue tracker and mirror of kubectl code
Apache License 2.0
2.82k stars 910 forks source link

`kubectl` `v1.29.3` ignores `HTTPS_PROXY` environment variable #1653

Closed kriswuollett closed 1 week ago

kriswuollett commented 1 week ago

What happened:

I am unable to connect to the API via a socks5 proxy via the HTTPS_PROXY environment variable. Instead it seems like only works when I add the same value to the proxy-url property in the cluster config.

What you expected to happen:

I expected to be able to use kubectl via a HTTPS proxy environment variable instead of the proxy-url attribute according to the documentation:

To access the Kubernetes API server through the proxy you must instruct kubectl to send queries through the SOCKS proxy we created earlier. Do this by either setting the appropriate environment variable, or via the proxy-url attribute in the kubeconfig file.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

The ability to use HTTPS_PROXY environment variable is extermely useful since it avoids hard-coding a port number to use for proxy access. The proxy-url attribute assumes one would want to statically assign different port numbers across numerous clusters your user account may wish to access.

Environment:

Client:

Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
macOS 14.6.1 (23G93)

Remote cluster:

kind version 0.24.0
debian 12 x86_64
ardaguclu commented 1 week ago

Interesting, could you please try;

HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-remote-host

also run this command with high verbosity to see;

HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-remote-host -v=9
ardaguclu commented 1 week ago

I can see that it is set by https://github.com/kubernetes/kubernetes/blob/3f3e1d507d081528ce8081f0daf4446d8a3b9bc0/staging/src/k8s.io/client-go/transport/cache.go#L125

kriswuollett commented 1 week ago

Why do I see curl in the log, shouldn't there be a native go HTTP client?

Without proxy-url attribute

% HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-nyc3-shared -v=9; kubectl version                   
I0912 19:59:50.425443   98668 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 19:59:50.430781   98668 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 19:59:50.432117   98668 round_trippers.go:508] HTTP Trace: Dial to tcp:127.0.0.1:34487 failed: dial tcp 127.0.0.1:34487: connect: connection refused
I0912 19:59:50.432133   98668 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500  in 1 milliseconds
I0912 19:59:50.432137   98668 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 1 ms
I0912 19:59:50.432140   98668 round_trippers.go:577] Response Headers:
I0912 19:59:50.432344   98668 helpers.go:264] Connection error: Get https://127.0.0.1:34487/api/v1/nodes?limit=500: dial tcp 127.0.0.1:34487: connect: connection refused
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?

Added proxy-url back to the context

It doesn't explictly say a proxy is being used other than the dial and curl port numbers are different.

% kubectl get nodes --context kind-nyc3-shared -v=9           
I0912 20:00:45.645722   98714 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 20:00:45.657230   98714 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 20:00:45.662387   98714 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost resolved to [{::1 } {127.0.0.1 }]
I0912 20:00:45.663187   98714 round_trippers.go:510] HTTP Trace: Dial to tcp:[::1]:8002 succeed
I0912 20:00:46.338098   98714 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500 200 OK in 680 milliseconds
I0912 20:00:46.338163   98714 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 233 ms ServerProcessing 223 ms Duration 680 ms
I0912 20:00:46.338175   98714 round_trippers.go:577] Response Headers:
I0912 20:00:46.338188   98714 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 0b411ff5-0cbc-4da2-b4c5-b8ca36074eb1
I0912 20:00:46.338197   98714 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 00fd2612-1189-46b0-b59b-2aa96bfc524b
I0912 20:00:46.338205   98714 round_trippers.go:580]     Date: Thu, 12 Sep 2024 12:00:45 GMT
I0912 20:00:46.338212   98714 round_trippers.go:580]     Audit-Id: 3f7f582a-753e-4b65-8059-88535c9b5cbe
I0912 20:00:46.338220   98714 round_trippers.go:580]     Cache-Control: no-cache, private
I0912 20:00:46.338228   98714 round_trippers.go:580]     Content-Type: application/json

...
ardaguclu commented 1 week ago

Sounds like an issue and requires investigation.

run a command like ssh -D 8002 -q -N remote-host to set up a socks proxy from localhost to remote-host

What is the exact command configuring this on Kind cluster?

ardaguclu commented 1 week ago

/triage accepted /priority backlog /assign

kriswuollett commented 1 week ago

There is no proxy information on the remote Kind cluster. Basically:

Don't think it is relevant, but here is my Kind config file:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: nyc3-shared
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

And I created it with the command kind create cluster --config /usr/local/etc/kind.yaml --wait 5m. Its running on a DigitalOcean droplet with debian-12-x64 image.

kriswuollett commented 1 week ago

I also tried on a Ubuntu 22.04 x86_64 host of mine:

$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.0

It has the same behavior as above on my mac. Also tried a more recent kubectl build with no success:

$ ./kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.0
ardaguclu commented 1 week ago

Thank you. I don't have remote host to test this and I need to find a way to reproduce this just on my local.

xyz-li commented 1 week ago

Why do I see curl in the log, shouldn't there be a native go HTTP client?

Without proxy-url attribute

% HTTPS_PROXY=socks5://localhost:8002; kubectl get nodes --context kind-nyc3-shared -v=9; kubectl version                   
I0912 19:59:50.425443   98668 loader.go:395] Config loaded from file:  /Users/kris/.kube/config
I0912 19:59:50.430781   98668 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.29.3 (darwin/arm64) kubernetes/6813625" 'https://127.0.0.1:34487/api/v1/nodes?limit=500'
I0912 19:59:50.432117   98668 round_trippers.go:508] HTTP Trace: Dial to tcp:127.0.0.1:34487 failed: dial tcp 127.0.0.1:34487: connect: connection refused
I0912 19:59:50.432133   98668 round_trippers.go:553] GET https://127.0.0.1:34487/api/v1/nodes?limit=500  in 1 milliseconds
I0912 19:59:50.432137   98668 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 1 ms
I0912 19:59:50.432140   98668 round_trippers.go:577] Response Headers:
I0912 19:59:50.432344   98668 helpers.go:264] Connection error: Get https://127.0.0.1:34487/api/v1/nodes?limit=500: dial tcp 127.0.0.1:34487: connect: connection refused
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server 127.0.0.1:34487 was refused - did you specify the right host or port?

If the apiserver address is https://127.0.0.1:34487, kubectl (specially go runtime) will ignore HTTPS_PROXY. According to the implementation of golang. It ignores HTTP_PROXY/HTTPS_PROXY when dest IP is localhost or loopbak. https://github.com/golang/go/blob/master/src/vendor/golang.org/x/net/http/httpproxy/proxy.go#L169-L202 I don't think this is a bug.

ardaguclu commented 1 week ago

Thanks @xyz-li. In that case how does proxy-url work?

kriswuollett commented 1 week ago

If kubectl has a command line option for things like insecure-skip-tls-verify, then why not add proxy-url as a kubectl command line parameter, which would override any setting in the config, to solve this issue? That way the whole issue of the environment variable usage not really being a standard can be dropped.

xyz-li commented 1 week ago

Thanks @xyz-li. In that case how does proxy-url work?

https://github.com/kubernetes/kubernetes/blob/359b9ba9bf5c508aeee0eee520d09b7bf1fdaa50/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L198-L206 @ardaguclu

ardaguclu commented 1 week ago

As there is no bug, I think we can close this issue. @kriswuollett the ask you above seems like a feature request and can be handled in a separate issue.

/close

k8s-ci-robot commented 1 week ago

@ardaguclu: Closing this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/1653#issuecomment-2348023010): >As there is no bug, I think we can close this issue. @kriswuollett the ask you above seems like a feature request and can be handled in a separate issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.