Open hirannair opened 3 years ago
Thanks @hirannair for opening this issue.
When we "refresh credentials", we actually delegate that operation to kubectl
:
kubectl get pods -n <YOUR_NAMESPACE>
).In the case you describe, it sounds like we're having troubles with the reply we get from kubectl
.
Would it be possible for you to run kubectl get pods -n <YOUR_NAMESPACE>
for the same cluster you use with Bridge to Kubernetes, and let me know what response you get?
Thanks @daniv-msft for that quick response.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fin-nginx-ingress-controller-58556b9589-ccj9b 1/1 Running 0 61d
fin-nginx-ingress-controller-58556b9589-qsb8m 1/1 Running 0 61d
fin-redis-cache-master-0 1/1 Running 0 61d
fin-redis-cache-slave-0 1/1 Running 4 61d
fin-redis-cache-slave-1 1/1 Running 2 51d
routingmanager-deployment-5d75f8cb77-ptg2r 1/1 Running 0 23h
Im able to get the list of pods through kubectl command line and kubernetes extension in vscode
Thanks @hirannair for your reply. I investigated more this issue, and it's possible that it comes from the version of the C# KubernetesClient library we use. We're using a version that isn't the latest, and it's possible this version is the one returning the BadGateway error in your case.
We're working on integrating with their latest version to determine if it helps with your case. Would you be willing to try beta bits of our extension including this change? I can provide full instructions once the fix is available (we're trying to get it done by tomorrow).
@daniv-msft Thanks for the response.Sure,Happy to test the beta version of the extension if you can provide the instructions on how to get it running
(+ @amsoedal who worked on increasing our dependency's version) @hirannair Thanks! We're working to provide this to you tomorrow.
Hi @hirannair, I've posted the steps here (https://github.com/microsoft/mindaro/issues/174#issuecomment-879313528) on how to test beta bits. Please let us know how it goes for you! :)
Thanks @amsoedal .Still getting the same error
below are the logs
2021-07-14T00:57:50.266Z | Common Extension Root | TRACE | query-expfeature <json>{"ABExp.queriedFeature":"vscode.mindaroBinariesVersion-1.0.20210702","remoteName":"dev-container"}</json>
2021-07-14T00:57:50.266Z | Common Extension Root | TRACE | Call to ExP returned version '1.0.20210708.15' <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.266Z | Common Extension Root | TRACE | Resolved expected CLI version '1.0.20210712.3' <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.286Z | Common Extension Root | TRACE | Event: binaries-utility-version-v1 <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.300Z | Common Extension Root | TRACE | Trying to initialize the workspace folder docker-dev for Connect <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.301Z | Common Extension Root | TRACE | Connect initialization started on docker-dev <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.305Z | Common Extension Root | TRACE | Event: connect-initialization-success <json>{"workspacesCommonId":"22404a8d-400c-7e38-a99b-abaa1cc52737","remoteName":"dev-container"}</json>
2021-07-14T00:57:50.316Z | Common Extension Root | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.345Z | Common Extension Root | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.368Z | Common Extension Root | WARNG | Binaries are not present locally: downloading them <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:50.387Z | Common Extension Root | TRACE | Resolved versionUrl 'https://mindarostaging.blob.core.windows.net/zip/1.0.20210712.3/lks.json' <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:57:51.734Z | Common Extension Root | TRACE | Event: cli-versions-client-get-download-info-success <json>{"availableVersion":"1.0.20210712.3","expectedVersion":"1.0.20210712.3","remoteName":"dev-container"}</json>
2021-07-14T00:57:51.735Z | Common Extension Root | TRACE | Event: cli-version-client-get-download-info-status <json>{"binariesVersionsDownloadTimeInMilliseconds":1361,"binariesVersionsDownloadSucceeded":true,"remoteName":"dev-contain
er"}</json>
2021-07-14T00:57:51.736Z | Common Extension Root | TRACE | Event: binaries-utility-download-start <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:05.247Z | Common Extension Root | TRACE | Event: configure-command-triggered <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:05.250Z | Connect (docker-dev/444d | TRACE | Event: connect-wizard-start <json>{"wizardReason":"ConfigurationCommand","type":"service","remoteName":"dev-container"}</json>
2021-07-14T00:58:05.254Z | Common Extension Root | WARNG | Event: binaries-utility-try-get-binaries-error <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:05.254Z | Common Extension Root | WARNG | Event: binaries-utility-try-get-binaries-error <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:05.255Z | Connect (docker-dev/444d | TRACE | Event: connect-wizard-stop <json>{"wizardReason":"ConfigurationCommand","type":"service","isWizardComplete":false,"isResourceNameSet":"false","isIsolateAsSet":"false","isTarg
etClusterSet":"false","isTargetNamespaceSet":"false","isCreatingNewLaunchConfiguration":false,"remoteName":"dev-container"}</json>
2021-07-14T00:58:18.043Z | Common Extension Root | TRACE | Making sure that the CLI is present locally, by downloading it if needed <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:46.835Z | Common Extension Root | TRACE | Event: binaries-utility-download-success <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:46.836Z | Common Extension Root | TRACE | Event: binaries-utility-overall-download-status <json>{"binariesDownloadTimeInMilliseconds":55100,"binariesDownloadSucceeded":true,"remoteName":"dev-container"}</json>
2021-07-14T00:58:50.243Z | Common Extension Root | TRACE | Event: cli-client-get-version-success <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:50.244Z | Common Extension Root | TRACE | Found local CLI version: '1.0.20210712.3'. Expected version: '1.0.20210712.3' <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:50.244Z | Common Extension Root | TRACE | Local CLI has version number '1.0.20210712.3', which is equal to the required '1.0.20210712.3' <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:50.245Z | Common Extension Root | TRACE | Event: binaries-utility-ensure-cli-success <json>{"isUsingLocalBinaries":"false","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.257Z | Common Extension Root | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.279Z | Common Extension Root | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.329Z | Common Extension Root | TRACE | Event: kubectl-client-command <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.642Z | Common Extension Root | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.644Z | Common Extension Root | TRACE | Event: kubectl-client-command <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.680Z | Common Extension Root | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.687Z | Common Extension Root | TRACE | Event: activation <json>{"workspacesCommonId":"22404a8d-400c-7e38-a99b-abaa1cc52737","workspaceFoldersCount":"1","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.687Z | Common Extension Root | TRACE | Extension activated successfully <json>{"remoteName":"dev-container"}</json>
2021-07-14T00:58:50.768Z | Common Extension Root | TRACE | Event: kubectl-client-command-success <json>{"args":"config view --minify -o json","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.769Z | Common Extension Root | TRACE | Event: kubectl-client-command <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.904Z | Common Extension Root | TRACE | Event: kubectl-client-command-success <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.904Z | Common Extension Root | TRACE | Event: kubernetes-panel-customizer-supported-fqdn-evaluation <json>{"currentFqdnDomain":"co.nz","clustersCount":4,"fqdnDomains":"co.nz,co.nz,co.nz,co.nz","remoteName":"dev-co
ntainer"}</json>
2021-07-14T00:58:50.992Z | Common Extension Root | TRACE | Event: kubectl-client-command-success <json>{"args":"config view -o jsonpath={.clusters[*].cluster.server}","remoteName":"dev-container"}</json>
2021-07-14T00:58:50.993Z | Common Extension Root | TRACE | Event: kubernetes-panel-customizer-supported-fqdn-evaluation <json>{"currentFqdnDomain":"co.nz","clustersCount":4,"fqdnDomains":"co.nz,co.nz,co.nz,co.nz","remoteName":"dev-co
ntainer"}</json>
2021-07-14T00:59:51.954Z | Common Extension Root | TRACE | Extension deactivation <json>{"remoteName":"dev-container"}</json>
Thanks @hirannair for giving it a try! I confirm you're running the right version of our binaries, so unfortunately it looks like this issue wasn't fixed in the latest version of our dependency. I will open a Github issue on their side to report that and see how we can proceed from here. Before I do, and to make sure I have enough information to open the issue on their repository, could you please reproduce the bug and attach the logs from this folder?
I'm interested in the latest bridge-library-*.txt
and bridge-mindarocli-*.txt
files, that should show the exact exception we get from the C# KubernetesClient.
Thanks for your help!
@daniv-msft Thanks for that quick response.Below are the logs
$ cat bridge-mindarocli-2021-07-14-01-14-13-1344.txt
2021-07-14T01:14:13.9329823Z | MindaroCli | TRACE | Event: Command.Start <json>{"properties":{"arguments":"refresh-credentials --namespace fin-acceptance","isRoutingEnabled":"false"},"metrics":null}</json>\nOperation context: <json>{"cl
ientRequestId":null,"correlationId":"b0464677-5cce-4795-9b1f-912f86f4ccc81626224689906:4559d60919c2","requestId":null,"userSubscriptionId":null,"startTime":"2021-07-14T01:14:13.5524405Z","userAgent":"VSCode/1.0.120210713","requestHttpMe
thod":null,"requestUri":null,"version":"1.0.20210712.3","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Linux 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021","framework":".NET Core
3.1.9","macAddressHash":"3424b72ee104e99be183dc60a7b751a058f3ff95747d04922fb59889f7e032b1","processId":1344,"targetEnvironment":"Staging","commandId":"4559d60919c2"}}</json>
2021-07-14T01:14:14.0424937Z | MindaroCli | TRACE | Running Microsoft.BridgeToKubernetes.Exe.Commands.Connect.RefreshCredentialsCommand...
2021-07-14T01:14:14.0459617Z | MindaroCli | TRACE | Initializing progress reporter...
2021-07-14T01:14:15.2041204Z | MindaroCli | ERROR | Dependency: List Contexts <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>
2021-07-14T01:14:15.2558809Z | MindaroCli | ERROR | Logging handled exception: Microsoft.Rest.HttpOperationException: {"Request":{"Method":{"Method":"GET"},"RequestUri":"https://REDACTED/api/v1/name
spaces/fin-acceptance/pods","Properties":{},"Content":null,"Headers":{"Authorization":["REDACTED"]}},"Response":{"StatusCode":502,"ReasonPhrase":"Bad Gateway","Content":"<html><head><title>502 Bad Gateway</title></head>\r\n<body
><h1>DNS error</h1>\r\n<p>DNS error (the host name of the page you are looking for does not exist)<br><br>Please check that the host name has been spelled correctly.<br></p>\r\n<!--Zscaler--></body></html>\r\n","Headers":{"Server":[
"Zscaler"],"Connection":["close"],"Content-Type":["text/html"]}},"Body":null,"StackTrace":" at k8s.Kubernetes.ListNamespacedPodWithHttpMessagesAsync(String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParamet
er, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellatio
nToken)\n at k8s.KubernetesExtensions.ListNamespacedPodAsync(IKubernetes operations, String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, St
ring resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass
29_0.<<ListPodsInNamespaceAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass75_0`1.<<ClientInvok
eWrapperAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func`2 action, TimeSpan maxWaitTime, Can
cellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func`1 handler, String operation, CancellationToken cancellationToken)\n at Microsoft.BridgeToKuberne
tes.Common.Kubernetes.KubernetesClient.ListPodsInNamespaceAsync(String namespaceName, IEnumerable`1 labels, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.Client.ManagementClients.KubernetesManagementCl
ient.<>c__DisplayClass6_0.<<RefreshCredentialsAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionS
trategy.<>c__DisplayClass3_0.<<RunWithHandlingAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionS
trategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n at Microsoft.BridgeToKubernetes.Library.Client.ManagementClients.KubernetesManagementClient.RefreshCredentialsAsync(IProgress`1 progress, String targetNamespa
ce, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.RefreshCredentialsCommand.ExecuteAsync()","Message":"Operation returned an invalid status code 'BadGateway'","Data":{},"InnerException":nu
ll,"HelpLink":null,"Source":"KubernetesClient","HResult":-2146233088}
2021-07-14T01:14:15.2583781Z | MindaroCli | ERROR | Operation returned an invalid status code 'BadGateway'\n
2021-07-14T01:14:15.6499073Z | MindaroCli | TRACE | Event: Command.End <json>{"properties":{"arguments":"refresh-credentials --namespace fin-acceptance","result":"Failed","failureReason":"Operation returned an invalid status code 'BadGa
teway'"},"metrics":{"duration":1848.0}}</json>
$ cat bridge-library-2021-07-14-01-14-14-1344.txt
2021-07-14T01:14:14.0887569Z | Library | TRACE | Pulling kubeconfig...\nOperation context: <json>{"clientRequestId":null,"correlationId":"b0464677-5cce-4795-9b1f-912f86f4ccc81626224689906:4559d60919c2:4261ee7f1ef5","requestId":null,"use
rSubscriptionId":null,"startTime":"2021-07-14T01:14:14.0054374Z","userAgent":"VSCode/1.0.120210713","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210712.3","requestHeaders":{},"loggingProperties":{"applicationName":"Librar
y","deviceOperatingSystem":"Linux 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"3424b72ee104e99be183dc60a7b751a058f3ff95747d04922fb59889f7e032b1","processId":1344,"targetEnvironmen
t":"Staging"}}</json>
2021-07-14T01:14:14.8171642Z | Library | TRACE | Event: CloudProvider <json>{"properties":{"clusterFQDNDomain":"co.nz"},"metrics":null}</json>
\n'!--Zscaler--></body></html>he page you are looking for does not exist)<br><br>Please check that the host name has been spelled correctly.<br></p>ay', Content='<html><head><title>502 Bad Gateway</title></head>
2021-07-14T01:14:15.0643992Z | Library | TRACE | Event: KubernetesManagementClient-RefreshCredentialsAsync <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":203.0}}</json>\nOperation context: <json>{"clientRequestId":"80
0b8158-9083-4d13-8d7b-314142509296","correlationId":"b0464677-5cce-4795-9b1f-912f86f4ccc81626224689906:4559d60919c2:972988002d1e","requestId":null,"userSubscriptionId":null,"startTime":"2021-07-14T01:14:14.0054374Z","userAgent":"VSCode/
1.0.120210713","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210712.3","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 202
1","framework":".NET Core 3.1.9","macAddressHash":"3424b72ee104e99be183dc60a7b751a058f3ff95747d04922fb59889f7e032b1","processId":1344,"targetEnvironment":"Staging"}}</json>
2021-07-14T01:14:15.0716929Z | Library | ERROR | Failed to Refresh Credentials
2021-07-14T01:14:15.1800658Z | Library | ERROR | Logging handled exception: Microsoft.Rest.HttpOperationException: {"Request":{"Method":{"Method":"GET"},"RequestUri":"https://REDACTED/api/v1/namespa
ces/fin-acceptance/pods","Properties":{},"Content":null,"Headers":{"Authorization":["REDACTED"]}},"Response":{"StatusCode":502,"ReasonPhrase":"Bad Gateway","Content":"<html><head><title>502 Bad Gateway</title></head>\r\n<body><h
1>DNS error</h1>\r\n<p>DNS error (the host name of the page you are looking for does not exist)<br><br>Please check that the host name has been spelled correctly.<br></p>\r\n<!--Zscaler--></body></html>\r\n","Headers":{"Server":["Zs
caler"],"Connection":["close"],"Content-Type":["text/html"]}},"Body":null,"StackTrace":" at k8s.Kubernetes.ListNamespacedPodWithHttpMessagesAsync(String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter,
String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationTo
ken)\n at k8s.KubernetesExtensions.ListNamespacedPodAsync(IKubernetes operations, String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, Strin
g resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass29_
0.<<ListPodsInNamespaceAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.<>c__DisplayClass75_0`1.<<ClientInvokeWr
apperAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func`2 action, TimeSpan maxWaitTime, Cancel
lationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func`1 handler, String operation, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes
.Common.Kubernetes.KubernetesClient.ListPodsInNamespaceAsync(String namespaceName, IEnumerable`1 labels, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.Client.ManagementClients.KubernetesManagementClien
t.<>c__DisplayClass6_0.<<RefreshCredentialsAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStra
tegy.<>c__DisplayClass3_0.<<RunWithHandlingAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStra
tegy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","Message":"Operation returned an invalid status code 'BadGateway'","Data":{},"InnerException":null,"HelpLink":null,"Source":"KubernetesClient","HResult":-2146233088
}
Thanks @hirannair! I logged this issue to ask for their help understanding what happens: https://github.com/kubernetes-client/csharp/issues/662
I can confirm that I do have the exact same problem. The problem seems indeed to be related to an enforced reverse proxy (ZScaler) here as well.
Describe the bug Im getting the below error when trying to configure Bridge to kubernetes
To Reproduce Install bridge to kubernetes extension in vscode and try to configure it
Ours is a private kubernetes cluster that runs in our datacentres, and we manage it.
Not sure why the extension is trying to refresh the token as we have a separate login process to get a new token.Our k8s token is a long lived token,so its not required to refresh it.
**Expected behavior** A clear and concise description of what you expected to happen. **Logs** ``` 2021-07-09T03:10:27.248Z | Common Extension Root | TRACE | Event: kubectl-client-command