search

microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

Operation returned an invalid status code 'Unauthorized' #174

Open jarrett-confrey opened 3 years ago

jarrett-confrey commented 3 years ago

Describe the bug

when i start vscode i see a spinner next to kubernetes in the status bar. after a couple seconds the spinner turns into a warning symbol. if i click on kubernetes in the status bar at this point i see Failed to refresh the kubeconfig token: Operation returned an invalid status code 'Unauthorized'.

OR

in the kubernetes cluster explorer when i right click on a pod and select Debug (Local Tunnel) i see the same Failed to refresh the kubeconfig token: Operation returned an invalid status code 'Unauthorized' at the bottom right of the screen. all the other functions in the pod right click menu appear to work fine. kubectl cli works fine. the npm module @kubernetes/client-node works fine. auth-provider is gangway/dex oidc

To Reproduce

start vscode and click on kubernetes in the status bar

OR

right click on a pod and select Debug (Local Tunnel).

Logs

2021-05-17T04:52:10.294Z | Common Extension Root    | TRACE | Event: binaries-utility-version-v1
2021-05-17T04:52:10.294Z | Common Extension Root    | TRACE | Trying to initialize the workspace folder MY_REPO for Connect
2021-05-17T04:52:10.295Z | Common Extension Root    | TRACE | Connect initialization started on MY_REPO
2021-05-17T04:52:10.295Z | Common Extension Root    | TRACE | Event: connect-initialization-success <json>{"workspacesCommonId":"08be87ee-345e-4569-1f10-88f068f4a4e9"}</json>
2021-05-17T04:52:10.296Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed
2021-05-17T04:52:10.297Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed
2021-05-17T04:52:10.309Z | Common Extension Root    | TRACE | Successfully determined the CLI binary path: /Users/MY_USER/Library/Application Support/Code/User/globalStorage/mindaro.mindaro/file-downloader-downloads/binaries/dsc
2021-05-17T04:52:10.318Z | Common Extension Root    | TRACE | Successfully determined the kubectl binary path: /Users/MY_USER/Library/Application Support/Code/User/globalStorage/mindaro.mindaro/file-downloader-downloads/binaries/kubectl/osx/kubectl
2021-05-17T04:52:12.403Z | Common Extension Root    | TRACE | Event: cli-client-get-version-success
2021-05-17T04:52:12.403Z | Common Extension Root    | TRACE | Found local CLI version: '1.0.20210507.2'. Minimum expected version: '1.0.20210506.1'
2021-05-17T04:52:12.403Z | Common Extension Root    | TRACE | Local CLI has version number '1.0.20210507.2', which is greater than or equal to the minimum requirement '1.0.20210506.1'
2021-05-17T04:52:12.403Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"version --short --client -o json"}</json>
2021-05-17T04:52:12.459Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"version --short --client -o json"}</json>
2021-05-17T04:52:12.459Z | Common Extension Root    | TRACE | Event: kubectl-client-get-version-success
2021-05-17T04:52:12.460Z | Common Extension Root    | TRACE | Event: binaries-utility-ensure-cli-success <json>{"isUsingLocalBinaries":"false"}</json>
2021-05-17T04:52:12.462Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:52:12.466Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:52:12.524Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:52:12.524Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}"}</json>
2021-05-17T04:52:12.528Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:52:12.581Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}"}</json>
2021-05-17T04:52:12.582Z | Common Extension Root    | TRACE | Event: kubernetes-panel-customizer-supported-fqdn-evaluation <json>{"currentFqdnDomain":"amazonaws.com","clustersCount":1,"fqdnDomains":"amazonaws.com"}</json>
2021-05-17T04:52:16.185Z | Common Extension Root    | ERROR | Error: cli-client-check-credentials-error <stack>Error: Operation returned an invalid status code 'Unauthorized'\n\n  at ChildProcess.<anonymous> (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:3:143268)\n  at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-05-17T04:52:16.186Z | Common Extension Root    | TRACE | Event: kubeconfig-credentials-manager-check-credentials-perf <json>{"success":false,"durationMs":3657}</json>
2021-05-17T04:52:16.186Z | Common Extension Root    | WARNG | Kubeconfig credentials need to be refreshed
2021-05-17T04:52:16.203Z | Common Extension Root    | TRACE | query-expfeature <json>{"ABExp.queriedFeature":"vscode.mindaroTreatmentVar"}</json>
2021-05-17T04:52:16.203Z | Common Extension Root    | TRACE | Event: activation <json>{"workspacesCommonId":"08be87ee-345e-4569-1f10-88f068f4a4e9","workspaceFoldersCount":"1"}</json>
2021-05-17T04:52:16.203Z | Common Extension Root    | TRACE | Extension activated successfully
2021-05-17T04:52:57.721Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed
2021-05-17T04:52:57.722Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:52:57.782Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-05-17T04:53:00.091Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-output-error <stack>SyntaxError: Unexpected token O in JSON at position 0\n at JSON.parse (<anonymous>)\n   at Array.<anonymous> (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:3:43449)\n  at t.EventSource.trigger (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:1:56357)\n  at Socket.<anonymous> (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:3:143089)\n    at Socket.emit (events.js:315:20)\n at addChunk (internal/streams/readable.js:309:12)\n at readableAddChunk (internal/streams/readable.js:284:9)\n  at Socket.Readable.push (internal/streams/readable.js:223:10)\n at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)</stack>
2021-05-17T04:53:01.651Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-error <stack>Error: Operation returned an invalid status code 'Unauthorized'\n\n    at ChildProcess.<anonymous> (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:3:143268)\n  at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-05-17T04:53:01.652Z | Common Extension Root    | ERROR | Error: kubeconfig-credentials-manager-refresh-credentials-error <stack>Error: Operation returned an invalid status code 'Unauthorized'\n\n    at ChildProcess.<anonymous> (/Users/MY_USER/.vscode/extensions/mindaro.mindaro-1.0.120210507/dist/extension.js:3:143268)\n  at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-05-17T04:53:01.652Z | Common Extension Root    | TRACE | Event: kubeconfig-credentials-manager-refresh-credentials-perf <json>{"success":false,"refreshRequired":false,"durationMs":3870}</json>
2021-05-17T04:53:01.653Z | Common Extension Root    | WARNG | Kubeconfig credentials need to be refreshed

Environment Details

Client used: VS Code Client's version: 1.0.120210507 Operating System: macOS 11.3.1 (20E241)

Additional context

im super excited to try out this feature, thanks for all the hard work!

amsoedal commented 3 years ago

@jarrett-confrey thanks for reaching out to us! Could you please attach the logs from /tmp/Bridge To Kubernetes (or send them at BridgeToKubernetes@microsoft.com) as well? That should help us in understanding what's going on

jarrett-confrey commented 3 years ago

@amsoedal i have emailed the relevant logs to BridgeToKubernetes@microsoft.com. please let me know if theres anything else i can do to help get the the bottom of the issue. thanks!

amsoedal commented 3 years ago

@jarrett-confrey thanks for sending those! Could you confirm that kubectl get po works for you? I know you said the kubernetes CLI is working, but what we're doing under the hood should be the same thing.

jarrett-confrey commented 3 years ago

@amsoedal sure thing, thanks for looking into this! yes kubectl get po works for me, i do have my namespace specified in my kube config though so its more equivalent to kubectl get po --namespace MY_NAMESPACE. i am able to successfully get pods using the node kubernetes client without needing to do anything special besides specifying my namespace as well. i do not have permission to access the default namespace if that matters at all

amsoedal commented 3 years ago

@jarrett-confrey thanks for the details! Because our tools rely on the c# kubernetes client, I've opened an issue to clarify with them whether it could be a dependency issue (https://github.com/kubernetes-client/csharp/issues/624). I'll keep investigating and keep you posted

jarrett-confrey commented 3 years ago

i was able to fix the issue with https://github.com/kubernetes-client/csharp/pull/633

amsoedal commented 3 years ago

@jarrett-confrey that's awesome! It'll take a little while for us to pull in the changes to the c# kubernetes client, but is your kubeconfig working now that the token is refreshed?

jarrett-confrey commented 3 years ago

@amsoedal the issue was that my token did not need to be refreshed, it has a long life and the check for expiration was incorrect. i did not get the refresh working but kubernetes calls such as listing pods are now working for me. ill have to double check but i suspect that im not able refresh the long lived token in my environment. i believe i have seen similar refresh issues with other kubernetes clients when my token expires

what does the timeline for this kind of change looks like?

amsoedal commented 3 years ago

Oh I see, makes sense. Thanks for clarifying. Once your PR is merged and they release a new version including your change, we should be able to release our version including that update within a few weeks.

jarrett-confrey commented 3 years ago

@amsoedal sounds good, thank you. it appears that Bridge to Kubernetes is close source with no change log. is there a way to know when the csharp kubernetes client has been bumped in your project and when that change has been released to the vscode marketplace? im happy to test any beta versions. thanks!

amsoedal commented 3 years ago

@jarrett-confrey yes, we're currently closed source (although this will hopefully change in a few months). I can update this thread when the change is complete :) thanks for your patience!

jarrett-confrey commented 3 years ago

@amsoedal i saw that there was a bridge to kubernetes update today but im still facing the issue. any chance kubernetes-client can be updated to contain my fix? thanks!

rakeshvanga commented 3 years ago

@jarrett-confrey The fix for kubernetes-client is not yet released in the recent update. It would be released in a week or two. Thanks!

jarrett-confrey commented 3 years ago

@rakeshvanga @amsoedal i see that there was a bridge to kubernetes update on 7/8/2021 but im still seeing the issue. can you confirm if the fix is in 1.0.120210709 or not? if not do you know when the fix might go out? im eager to try out this plugin. thanks!

amsoedal commented 3 years ago

Hi @jarrett-confrey, I'm so sorry for the long wait. I just made the change to bump the package and it should go out as part of our next release. If you prefer, I can email you with some beta bits as soon as this gets checked in. Thanks so much for your patience!

jarrett-confrey commented 3 years ago

hey @amsoedal a wouldnt mind testing out a beta, thanks!

amsoedal commented 3 years ago

Hi @jarrett-confrey, these are the steps:

Let me know how it works for you! Thanks!

jarrett-confrey commented 3 years ago

@amsoedal i finally got a chance to test this but i am seeing the same issue. are you sure that this is using the correct kubernetes cshap client version that includes my fix?

amsoedal commented 3 years ago

@jarrett-confrey did you test using the steps above or using the latest extension? The staging bits should have them, but for the latest extension the very latest changes have only been flighted to 25% of users so far.