search

int128 / kubelogin

kubectl plugin for Kubernetes OpenID Connect authentication (kubectl oidc-login)
Apache License 2.0
1.68k stars 192 forks source link

Kubelogin hangs and spams indefinitely on macOS Ventura M1 #831

Open yafanasiev opened 1 year ago

yafanasiev commented 1 year ago

Describe the issue

When trying to run any command with kubelogin kubectl context (or even switch to it), I receive a bunch of error messages and whole system eventually hangs.

To reproduce

Example kubectl config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <REDACTED>
    server: <REDACTED>
  name: 
contexts:
- context:
    cluster: test-cluster
    user: test-cluster
  name: test-cluster
current-context: test-cluster
kind: Config
preferences: {}
users:
- name: test-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https:// <REDACTED>.okta.com
      - --oidc-client-id= <REDACTED>
      - --oidc-extra-scope=groups
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

Trying to run any kubectl command with this config will result in the following:

W1118 17:24:21.496477   73799 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.515760   73800 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.534581   73801 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.551069   73802 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.569885   73803 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.586530   73804 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.604317   73805 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.621143   73806 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.640124   73807 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.657907   73808 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.675308   73809 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.692730   73810 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.709909   73811 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.727902   73812 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.745933   73813 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.763129   73814 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.782300   73815 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.800037   73816 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.819771   73817 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.837103   73819 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.890598   73820 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.883850   73821 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.908978   73823 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.908940   73822 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.929781   73824 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.943233   73825 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.960170   73826 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.977258   73827 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.994883   73828 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.011367   73829 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.030513   73830 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.046929   73831 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.064543   73832 transport.go:243] Unable to cancel request for *exec.roundTripper

These errors appear very quickly and in a minute will hang the entire system unless terminal is closed first. Config is valid and is being used by at least a dozen of other team members (both Intel and M1 macs).

All cluster used for testing are AWS EKS 1.23. Tried installing kubelogin multiple different ways (brew, binary, krew, previous patch version, even x86_64 build through Rosetta) - result is the same.

Running kubectl oidc-login setup in another kubectl context works. Port 8000 is free.

What's most peculiar is that with this config

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <REDACTED>
    server: <REDACTED>
  name:
contexts:
- context:
    cluster: test-cluster
    user: test-cluster-aws
  name: test-cluster
current-context: test-cluster
kind: Config
preferences: {}
users:
- name: test-cluster-aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      -  <REDACTED>
      - eks
      - get-token
      - --cluster-name
      -  <REDACTED>
      command: aws
      env:
      - name: AWS_PROFILE
        value:  <REDACTED>
      interactiveMode: IfAvailable
      provideClusterInfo: false
- name: test-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https:// <REDACTED>.okta.com
      - --oidc-client-id= <REDACTED>
      - --oidc-extra-scope=groups
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

(default user is now aws cli exec) running kubectl --user=oidc get nodes works without issues:

I1118 17:33:38.186657   80446 get_token.go:51] WARNING: log may contain your secrets such as token or password
I1118 17:33:38.186765   80446 get_token.go:58] acquiring a lock get-token-8000-18000
I1118 17:33:38.187053   80446 get_token.go:70] finding a token from cache directory /Users/ <REDACTED>/.kube/cache/oidc-login
I1118 17:33:38.187245   80446 authentication.go:74] checking expiration of the existing token
I1118 17:33:38.187282   80446 authentication.go:83] you already have a valid token until 2022-11-18 18:04:50 +0200 EET
I1118 17:33:38.187309   80446 get_token.go:102] you got a token: { <REDACTED>}
I1118 17:33:38.187317   80446 get_token.go:105] you already have a valid token until 2022-11-18 18:04:50 +0200 EET
I1118 17:33:38.187322   80446 get_token.go:112] writing the token to client-go

but as soon as I switch user in kubectl context back to oidc auth it breaks again.

Your environment

TBH this drives me crazy for the last couple of days, any help at this point would be lifesaving

int128 commented 1 year ago

I'm not sure but this issue may be related to Go. I will release a new version built with Go 1.19.3.

yafanasiev commented 1 year ago

@int128 thanks a lot for the swift reply! So this is a little embarrassing, but I did more tinkering with the new version and eventually found out that my kubectl binary was no the one from Homebrew, but actually from Rancher Desktop installation. As soon as I removed it from the PATH and made sure kubectl binary was loaded from Homebrew installation (now version 1.25.4), everything started working as expected, both on kubelogin version 1.25.3 and 1.25.4, Homebrew and binary installation alike. It appears this is a known issue: https://github.com/rancher-sandbox/rancher-desktop/issues/1308. Sorry for the false alarm, since the issue is most likely on RD side. Thanks again!

kingdonb commented 1 year ago

Thank you @yafanasiev for posting this, I'm sure I wouldn't have found it for at least another hour without your help here 🎉

I removed the rancher desktop kubectl and ensured I am using a proper kubectl from homebrew, now that is the last piece in the puzzle for me so I can use kubelogin with my github org 🥇

Oddly enough I did not have problems until I got to the very last step. When setting the user of the context to oidc with --user=oidc at runtime, the rancher desktop kubectl had no issue for me. I'm sure I don't need that kubectl, so 😅

Thanks for leaving this issue open in spite of the root cause being somewhere else, I'm sure that helps with the visibility also.

fabioluissilva commented 9 months ago

I confirm that I had the same exact issue, removing Rancher Desktop and its specific kubectl (reverting to using homebrew's kubectl), everything works properly. I believe you can close this.

adamzachyang commented 7 months ago

I've recently ran into this issue of running the kubectlcommand resulting in the following as mentioned above trying to connect to an EKS environment on a M2 Pro Macbook Pro:

Trying to run any kubectl command with this config will result in the following:

W1118 17:24:21.496477   73799 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.515760   73800 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.534581   73801 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.551069   73802 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.569885   73803 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.586530   73804 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.604317   73805 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.621143   73806 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.640124   73807 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.657907   73808 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.675308   73809 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.692730   73810 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.709909   73811 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.727902   73812 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.745933   73813 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.763129   73814 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.782300   73815 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.800037   73816 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.819771   73817 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.837103   73819 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.890598   73820 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.883850   73821 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.908978   73823 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.908940   73822 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.929781   73824 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.943233   73825 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.960170   73826 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.977258   73827 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:21.994883   73828 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.011367   73829 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.030513   73830 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.046929   73831 transport.go:243] Unable to cancel request for *exec.roundTripper
W1118 17:24:22.064543   73832 transport.go:243] Unable to cancel request for *exec.roundTripper

These errors appear very quickly and in a minute will hang the entire system unless terminal is closed first. Config is valid and is being used by at least a dozen of other team members (both Intel and M1 macs).

All cluster used for testing are AWS EKS 1.23. Tried installing kubelogin multiple different ways (brew, binary, krew, previous patch version, even x86_64 build through Rosetta) - result is the same.

What seems to have caused the issue was that I had installed the intel compiled version of Rancher Desktop. Completely removing it and installing the Apple Silicon Rancher Desktop solved this issue completely!

kingdonb commented 7 months ago

Aha the architecture mismatch is something I had not considered. I haven't been using Rancher Desktop but I'll try it again today to confirm for myself - I am using an M1 mac so it's entirely possible I had the wrong architecture binary installed too.

I'm still using kubelogin and I only had this problem with .rd/bin in my path. It wasn't necessary to uninstall anything, but again, I had not thought about the possibility of an architecture mismatch, and I guess everything else was working – rosetta is pretty seamless and hard to notice? I have completely wiped my user since then, but it's good to get confirmation on the details.

farcop commented 6 months ago

I suppose this issue not platform dependant, it reproduces even at linux/amd64 env.

bck01215 commented 2 months ago

Also affected and removing .rd/bin from path fixed this on M1 mac

tjamet commented 1 month ago

Hi! Depending on how you installed the kubectl command, it can be related to this. Indeed, some distributions like rancher desktop does not install the actual kubectl but a wrapper kuberlr that is responsible for fetching the relevant kubectl for your cluster.

The problem is that this wrapper replacing kubectl will fetch the version from the server using the kubernetes go glient, which in its turn uses the kubectl command to authenticate, and hence fetch the version from the server, authenticating with kubectl, you got the recursion loop I guess.

I sense your issues are related to this specific problem. I suggested a fix to the author of the tool. He has been pretty prompt to react to it. So, hopefully, in the coming revisions this will be fixed

alekc commented 1 week ago

https://github.com/flavio/kuberlr/releases/tag/v0.5.2 fixes this issue.

Just replace kuberlr inside /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/bin