Failed to refresh the kubeconfig token: Internal error

Miles-Garnsey commented 3 years ago

Describe the bug

When I try to connect to the Kubernetes cluster by clicking the icon in the status bar, I get the following error Failed to refresh the kubeconfig token: Internal error.

To Reproduce

This has only just started happening. It appears to have started when I closed the editor while in a debugging session. I have restarted my machine since then, uninstalled and reinstalled the extension and the problem still persists.

Miles-Garnsey commented 3 years ago

Here is some log output -

2021-08-18T08:29:20.545Z | Common Extension Root    | TRACE | query-expfeature <json>{"ABExp.queriedFeature":"vscode.mindaroBinariesVersion-1.0.20210723"}</json>
2021-08-18T08:29:20.545Z | Common Extension Root    | TRACE | Call to ExP returned version '1.0.20210723.6' <json>{}</json>
2021-08-18T08:29:20.545Z | Common Extension Root    | TRACE | Resolved expected CLI version '1.0.20210723.6' <json>{}</json>
2021-08-18T08:29:20.547Z | Common Extension Root    | TRACE | Event: binaries-utility-version-v2 <json>{}</json>
2021-08-18T08:29:20.548Z | Common Extension Root    | TRACE | Trying to initialize the workspace folder k8ssandra-operator for Connect <json>{}</json>
2021-08-18T08:29:20.548Z | Common Extension Root    | TRACE | Connect initialization started on k8ssandra-operator <json>{}</json>
2021-08-18T08:29:20.549Z | Common Extension Root    | TRACE | Event: connect-initialization-success <json>{"workspacesCommonId":"4cb0a387-6361-aa12-282f-17c4fb124478"}</json>
2021-08-18T08:29:20.549Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{}</json>
2021-08-18T08:29:20.552Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{}</json>
2021-08-18T08:29:20.557Z | Common Extension Root    | TRACE | Successfully determined the dotnetruntime binary path: /Users/milesgarnsey/Library/Application Support/Code/User/globalStorage/mindaro.mindaro/file-downloader-downloads/dotnet/dotnet <json>{}</json>
2021-08-18T08:29:20.570Z | Common Extension Root    | TRACE | Event: dotnet-client-get-version-success <json>{}</json>
2021-08-18T08:29:20.570Z | Common Extension Root    | TRACE | Found local dotnetruntime version: '3.1.6'. Minimum expected version: '3.1.6' <json>{}</json>
2021-08-18T08:29:20.570Z | Common Extension Root    | TRACE | Local dotnetruntime has version number '3.1.6', which is greater than or equal to the minimum requirement '3.1.6' <json>{}</json>
2021-08-18T08:29:20.575Z | Common Extension Root    | TRACE | Successfully determined the bridge binary path: /Users/milesgarnsey/Library/Application Support/Code/User/globalStorage/mindaro.mindaro/file-downloader-downloads/bridge/bridge <json>{}</json>
2021-08-18T08:29:20.582Z | Common Extension Root    | TRACE | Successfully determined the kubectl binary path: /Users/milesgarnsey/Library/Application Support/Code/User/globalStorage/mindaro.mindaro/file-downloader-downloads/kubectl/osx/kubectl <json>{}</json>
2021-08-18T08:29:20.583Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"version --short --client -o json"}</json>
2021-08-18T08:29:20.709Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"version --short --client -o json"}</json>
2021-08-18T08:29:20.710Z | Common Extension Root    | TRACE | Event: kubectl-client-get-version-success <json>{}</json>
2021-08-18T08:29:20.710Z | Common Extension Root    | TRACE | Found local kubectl version: '1.21.2'. Minimum expected version: '1.21.2' <json>{}</json>
2021-08-18T08:29:20.710Z | Common Extension Root    | TRACE | Local kubectl has version number '1.21.2', which is greater than or equal to the minimum requirement '1.21.2' <json>{}</json>
2021-08-18T08:29:22.761Z | Common Extension Root    | TRACE | Event: cli-client-get-version-success <json>{}</json>
2021-08-18T08:29:22.761Z | Common Extension Root    | TRACE | Found local bridge version: '1.0.20210723.6'. Minimum expected version: '1.0.20210723.6' <json>{}</json>
2021-08-18T08:29:22.761Z | Common Extension Root    | TRACE | Local bridge has version number '1.0.20210723.6', which is greater than or equal to the minimum requirement '1.0.20210723.6' <json>{}</json>
2021-08-18T08:29:22.762Z | Common Extension Root    | TRACE | Event: binaries-utility-overall-download-status <json>{"numberOfBinariesDownloaded":0,"binariesDownloadTimeInMilliseconds":0,"binariesDownloadSucceeded":true}</json>
2021-08-18T08:29:22.763Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"version --short --client -o json"}</json>
2021-08-18T08:29:22.830Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"version --short --client -o json"}</json>
2021-08-18T08:29:22.830Z | Common Extension Root    | TRACE | Event: kubectl-client-get-version-success <json>{}</json>
2021-08-18T08:29:22.830Z | Common Extension Root    | TRACE | Found local kubectl version: '1.21.2'. Minimum expected version: '1.21.2' <json>{}</json>
2021-08-18T08:29:22.830Z | Common Extension Root    | TRACE | Local kubectl has version number '1.21.2', which is greater than or equal to the minimum requirement '1.21.2' <json>{}</json>
2021-08-18T08:29:22.831Z | Common Extension Root    | TRACE | Event: binaries-utility-ensure-binaries-success <json>{"isUsingLocalBinaries":"false","isUsingLocalDotNet":"false"}</json>
2021-08-18T08:29:22.833Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:22.838Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:22.907Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:22.907Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}"}</json>
2021-08-18T08:29:22.911Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:22.976Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}"}</json>
2021-08-18T08:29:22.976Z | Common Extension Root    | TRACE | Event: kubernetes-panel-customizer-supported-fqdn-evaluation <json>{"currentFqdnDomain":"0.1","clustersCount":1,"fqdnDomains":"0.1"}</json>
2021-08-18T08:29:26.347Z | Common Extension Root    | ERROR | Error: cli-client-check-credentials-error <json>{}</json> <stack>Error: Internal error\n\n    at ChildProcess.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144421)\n at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-08-18T08:29:26.348Z | Common Extension Root    | TRACE | Event: kubeconfig-credentials-manager-check-credentials-perf <json>{"success":false,"durationMs":3436}</json>
2021-08-18T08:29:26.348Z | Common Extension Root    | WARNG | Kubeconfig credentials need to be refreshed <json>{}</json>
2021-08-18T08:29:26.348Z | Common Extension Root    | TRACE | Event: activation <json>{"workspacesCommonId":"4cb0a387-6361-aa12-282f-17c4fb124478","workspaceFoldersCount":"1"}</json>
2021-08-18T08:29:26.348Z | Common Extension Root    | TRACE | Extension activated successfully <json>{}</json>
2021-08-18T08:29:29.586Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{}</json>
2021-08-18T08:29:29.586Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:29.647Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-08-18T08:29:31.075Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-output-error <json>{}</json> <stack>SyntaxError: Unexpected token I in JSON at position 0\n at JSON.parse (<anonymous>)\n   at Array.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:43490)\n at t.EventSource.trigger (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:1:57709)\n at Socket.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144242)\n   at Socket.emit (events.js:315:20)\n at addChunk (internal/streams/readable.js:309:12)\n at readableAddChunk (internal/streams/readable.js:284:9)\n  at Socket.Readable.push (internal/streams/readable.js:223:10)\n at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)</stack>
2021-08-18T08:29:32.491Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-error <json>{}</json> <stack>Error: Internal error\n\n  at ChildProcess.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144421)\n at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-08-18T08:29:32.492Z | Common Extension Root    | ERROR | Error: kubeconfig-credentials-manager-refresh-credentials-error <json>{}</json> <stack>Error: Internal error\n\n  at ChildProcess.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144421)\n at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-08-18T08:29:32.492Z | Common Extension Root    | TRACE | Event: kubeconfig-credentials-manager-refresh-credentials-perf <json>{"success":false,"refreshRequired":false,"durationMs":2845}</json>
2021-08-18T08:29:32.494Z | Common Extension Root    | WARNG | Kubeconfig credentials need to be refreshed <json>{}</json>
2021-08-18T08:35:09.253Z | Common Extension Root    | TRACE | Event: account-context-manager-kubeconfig-change <json>{}</json>
2021-08-18T09:02:02.547Z | Common Extension Root    | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{}</json>
2021-08-18T09:02:02.548Z | Common Extension Root    | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json"}</json>
2021-08-18T09:02:02.711Z | Common Extension Root    | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json"}</json>
2021-08-18T09:02:04.286Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-output-error <json>{}</json> <stack>SyntaxError: Unexpected token I in JSON at position 0\n at JSON.parse (<anonymous>)\n   at Array.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:43490)\n at t.EventSource.trigger (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:1:57709)\n at Socket.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144242)\n   at Socket.emit (events.js:315:20)\n at addChunk (internal/streams/readable.js:309:12)\n at readableAddChunk (internal/streams/readable.js:284:9)\n  at Socket.Readable.push (internal/streams/readable.js:223:10)\n at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)</stack>
2021-08-18T09:02:05.502Z | Common Extension Root    | ERROR | Error: cli-client-refresh-credentials-error <json>{}</json> <stack>Error: Internal error\n\n  at ChildProcess.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144421)\n at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-08-18T09:02:05.504Z | Common Extension Root    | ERROR | Error: kubeconfig-credentials-manager-refresh-credentials-error <json>{}</json> <stack>Error: Internal error\n\n  at ChildProcess.<anonymous> (/Users/milesgarnsey/.vscode/extensions/mindaro.mindaro-1.0.120210803/dist/extension.js:3:144421)\n at ChildProcess.emit (events.js:315:20)\n   at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)</stack>
2021-08-18T09:02:05.504Z | Common Extension Root    | TRACE | Event: kubeconfig-credentials-manager-refresh-credentials-perf <json>{"success":false,"refreshRequired":false,"durationMs":2793}</json>
2021-08-18T09:02:05.505Z | Common Extension Root    | WARNG | Kubeconfig credentials need to be refreshed <json>{}</json>

amsoedal commented 3 years ago

Hi @Miles-Garnsey, we're investigating the issue. Thanks for reporting.

amsoedal commented 3 years ago

Hi @Miles-Garnsey, I implemented a workaround and built a beta extension that uses it. Can you follow the steps here to use the beta extension and let me know if it unblocks you? Here are the instructions: https://github.com/microsoft/mindaro/issues/195#issuecomment-872663727

Miles-Garnsey commented 3 years ago

Hey @amsoedal thanks so much for coming back to me so quickly! Can I build this from source (for MacOS) somehow? I'm not sure if I'm allowed to download from Azure directly 🙁

(EDIT: if you just want to share the branch I can probably figure out how to build, but any additional tips would also be amazing!)

daniv-msft commented 3 years ago

Thanks @Miles-Garnsey for your reply! Unfortunately, Bridge to Kubernetes is not open source (yet!) so it's not possible for you to build it yourself. When you say you might not be allowed to download from Azure, do you mean from a firewall perspective? We might be able to send you the staging VSIX to use through another channel if that helps.

I don't expect that we'll do a production release before a couple of weeks, so ideally we'd like to unblock you before that.

Miles-Garnsey commented 3 years ago

@daniv-msft can you get it hosted on a site only MS can upload to? We're not sure about working with *.core.windows.net domains because of this issue.

I can probably wait for the release if necessary, but I'd love to help you folks test (and also get back to using BTK!)

daniv-msft commented 3 years ago

@Miles-Garnsey Thanks for clarifying! Unfortunately I don't think I have an option more secure than this. While I don't have much context about the issue you mention, the Microsoft security teams are usually super responsive to fix issues, and I couldn't believe this exploit could still be ongoing.

We've found a way to reproduce this Internal error issue internally, so we're pretty confident in the fix we have and would feel good pushing this with our next production release in a couple of weeks. However, I want to mention that even in production, in VS Code we do call the *.core.windows.net domain before retrieving the latest version of our binaries. Again, I want to reiterate that this is a secure solution and that the security of our product is audited internally regularly, but I want to mention this as you expressed concerns about this domain.

Miles-Garnsey commented 3 years ago

I can check the signature instead. Just figuring out how to do that, leave it with me.

Miles-Garnsey commented 3 years ago

OK, it doesn't seem to be signed. I don't think I can roll this out at the moment, but I'm really looking forward to the final release!

Miles-Garnsey commented 2 years ago

Hi, just looking for an update on this issue. It's been 7 months...

microsoft / mindaro

Failed to refresh the kubeconfig token: Internal error #220