launch in VSCode fails with "Failed to get elevation requests."

[sgandon]

Describe the bug I am trying to run the bridge on VSCode on my mac with a Go app that is deployed in a local cluster (k3d). I had to create a service to the pod because the pod does not require one, it is an autonomous slack bot. When launching the app I have the following error in the terminal

> Executing task: bridge-to-kubernetes.service <

Redirecting Kubernetes service botkube to your machine...
Current cluster: k3d-k3s-default
Current namespace: botkube
Target service name: botkube
Target service ports: 42530

Error: Oops... An unexpected error has occurred.
For diagnostic information, see logs at '/var/folders/xl/b6750v1n1jj0mk5cshtsx6yr0000gn/T/Bridge To Kubernetes'.

The terminal process terminated with exit code: 1.

Digging into the logs I have the following logs in the bridge-library-2020-10-29-17-54-35-58055.txt file.

2020-10-29T18:02:34.7148630Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-GetReachableServices <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":113.0}}</json>\nOperation context: <json>{"clientRequestId":null,"correlationRequestId":null,"requestId":null,"userSubscriptionId":null,"startTime":"2020-10-29T18:02:34.3791980Z","userAgent":"VSCode/1.0.120201021","requestHttpMethod":null,"requestUri":null,"version":"1.0.20201021.18","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Darwin 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64","framework":".NET Core 3.1.9","macAddressHash":"2442ea78560a87847acfad6c95c34961167425a202a59048528fe2d40d510c44","processId":58922,"targetEnvironment":"Production","isRoutingEnabled":false}}</json>
2020-10-29T18:02:34.7201280Z | Library | TRACE | Event: ConnectManagementClient-GetElevationRequests <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":137.0}}</json>
2020-10-29T18:02:34.7207530Z | Library | ERROR | Failed to get elevation requests.
2020-10-29T18:02:34.8762830Z | Library | ERROR | Logging handled exception: System.Net.Http.HttpRequestException: {"StackTrace":"   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n   at k8s.Kubernetes.ListNamespacedServiceWithHttpMessagesAsync(String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)\n   at k8s.KubernetesExtensions.ListNamespacedServiceAsync(IKubernetes operations, String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)\n   at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.<>c__DisplayClass30_0.<<ListServicesInNamespaceAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.<>c__DisplayClass47_0`1.<<ClientInvokeWrapperAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.DevSpaces.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func`2 action, TimeSpan maxWaitTime, CancellationToken cancellationToken)\n   at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func`1 handler, String operation, CancellationToken cancellationToken)\n   at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.ListServicesInNamespaceAsync(String namespaceName, IEnumerable`1 labels, CancellationToken cancellationToken)\n   at Microsoft.DevSpaces.Library.Connect.KubernetesRemoteEnvironmentManager.GetReachableServicesAsync(RemoteContainerConnectionDetails remoteContainerConnectionDetails, ILocalProcessConfig localProcessConfig, IProgress`1 progress, CancellationToken cancellationToken)\n   at Microsoft.DevSpaces.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass15_0.<<GetElevationRequestsAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.DevSpaces.Library.ManagementClients.DevSpacesManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","Message":"IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')","Data":{},"InnerException":{"ClassName":"System.ArgumentException","Message":"IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Net.Dns.HostResolutionBeginHelper(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, AsyncCallback requestCallback, Object state)\n   at System.Net.Dns.BeginGetHostAddresses(String hostNameOrAddress, AsyncCallback requestCallback, Object state)\n   at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(SocketAsyncEventArgs args, DnsEndPoint endPoint)\n   at System.Net.Sockets.Socket.ConnectAsync(SocketType socketType, ProtocolType protocolType, SocketAsyncEventArgs e)\n   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024809,"Source":"System.Net.NameResolution","WatsonBuckets":null,"ParamName":"hostName"},"HelpLink":null,"Source":"System.Net.Http","HResult":-2147024809}

To Reproduce install botkube with helm, create a service because it does not come with one

apiVersion: v1
kind: Service
metadata:
 name: botkube
 labels:
   app: botkube
spec:
 type: ClusterIP
 ports:
 - name: "foo"
   port: 1234
 selector:
   app: botkube

create a config in VSCode with 0 port to forward and no isolation. run the main.go "Launch with Kubernetes"

**Expected behavior** A clear and concise description of what you expected to happen. **Logs** Attach logs from the following directory: For Windows: %TEMP%/Bridge to Kubernetes For OSX/Linux: $TMPDIR/Bridge to Kubernetes this is the bridge-mindarocli.txt log ``` 2020-10-29T18:02:34.3174350Z | MindaroCli | TRACE | Event: Command.Start {"properties":{"arguments":"prep-connect --output json","isRoutingEnabled":"false"},"metrics":null}\nOperation context: {"clientRequestId":null,"correlationRequestId":null,"requestId":null,"userSubscriptionId":null,"startTime":"2020-10-29T18:02:33.8955560Z","userAgent":"VSCode/1.0.120201021","requestHttpMethod":null,"requestUri":null,"version":"1.0.20201021.18","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Darwin 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64","framework":".NET Core 3.1.9","macAddressHash":"2442ea78560a87847acfad6c95c34961167425a202a59048528fe2d40d510c44","processId":58922,"targetEnvironment":"Production","commandId":"f3fe95c1-a685-447f-98ac-a0076cd68b2f"}} 2020-10-29T18:02:34.8906600Z | MindaroCli | ERROR | Dependency: Prep Connect {"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}} 2020-10-29T18:02:34.8980190Z | MindaroCli | ERROR | Oops... An unexpected error has occurred.\n 2020-10-29T18:02:34.8990740Z | MindaroCli | ERROR | For diagnostic information, see logs at '/var/folders/xl/b6750v1n1jj0mk5cshtsx6yr0000gn/T/Bridge To Kubernetes'.\n 2020-10-29T18:02:34.9134100Z | MindaroCli | ERROR | Logging handled exception: System.Net.Http.HttpRequestException: {"StackTrace":" at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n at k8s.Kubernetes.ListNamespacedServiceWithHttpMessagesAsync(String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)\n at k8s.KubernetesExtensions.ListNamespacedServiceAsync(IKubernetes operations, String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.<>c__DisplayClass30_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.<>c__DisplayClass47_0`1.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.DevSpaces.Common.Utilities.WebUtilities.RetryUntilTimeAsync(Func`2 action, TimeSpan maxWaitTime, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.ClientInvokeWrapperAsync[T](Func`1 handler, String operation, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Common.Kubernetes.KubernetesClient.ListServicesInNamespaceAsync(String namespaceName, IEnumerable`1 labels, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Library.Connect.KubernetesRemoteEnvironmentManager.GetReachableServicesAsync(RemoteContainerConnectionDetails remoteContainerConnectionDetails, ILocalProcessConfig localProcessConfig, IProgress`1 progress, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass15_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.DevSpaces.Library.ManagementClients.DevSpacesManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n at Microsoft.DevSpaces.Library.ManagementClients.ConnectManagementClient.GetElevationRequestsAsync(IProgress`1 progress, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Exe.Commands.Connect.PrepConnectCommand.ExecuteAsync()\n at Microsoft.DevSpaces.Exe.DevSpacesCliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\n at Microsoft.DevSpaces.Exe.DevSpacesCliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","Message":"IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')","Data":{},"InnerException":{"ClassName":"System.ArgumentException","Message":"IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Net.Dns.HostResolutionBeginHelper(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, AsyncCallback requestCallback, Object state)\n at System.Net.Dns.BeginGetHostAddresses(String hostNameOrAddress, AsyncCallback requestCallback, Object state)\n at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(SocketAsyncEventArgs args, DnsEndPoint endPoint)\n at System.Net.Sockets.Socket.ConnectAsync(SocketType socketType, ProtocolType protocolType, SocketAsyncEventArgs e)\n at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024809,"Source":"System.Net.NameResolution","WatsonBuckets":null,"ParamName":"hostName"},"HelpLink":null,"Source":"System.Net.Http","HResult":-2147024809} 2020-10-29T18:02:34.9147990Z | MindaroCli | TRACE | Event: Command.End {"properties":{"arguments":"prep-connect --output json","result":"Failed"},"metrics":{"duration":672.0}} ``` **Environment Details** Client used (VS Code): ``` code --version 1.50.1 d2e414d9e4239a252d1ab117bd7067f125afd80a x64 ``` ` mindaro.mindaro@1.0.120201021` Client's version: Operating System: OSX 10.15.7 **Additional context** Add any other outputs from the clients or context you would like to share.

[greenie-msft]

Hi @sgandon

Thank you for reporting this issue. I'm logging a work-item for our engineering team to investigate. I will post back to thread as soon as we have an update to share.

Thank you.

[amsoedal]

@sgandon, could you please send us the full log files? You can email it to me at ansoedal@microsoft.com if you feel most comfortable that way. Thanks!

[sgandon]

@amsoedal I am not sure what king of other log I can provide, I only found 2 kind of files in the log folder and they are both (bridge-mindarocli.txt and bridge-library.xxxx.txt) displayed fully in the issue description. I will zip the complete folder and send it to you.

[faysou]

I got exactly the same error, while using Openshift CRC as local cluster.

[amsoedal]

Hi @faysou thanks for reporting this! Would you be able to upload/send your logs to BridgeToKubernetes@microsoft.com as well?

The logs can be found at: For Windows: %TEMP%/Bridge to Kubernetes For OSX/Linux: $TMPDIR/Bridge to Kubernetes

[faysou]

Thank you. I got the problem while debugging a Go app as well, I don't know if it's related, I'll try with another language to test.

I've sent you the two log files. Fingers crossed you can do something, I'm eager to test this extension, it seems amazing to accelerate developing microservices. On top of that it seems to be a language agnostic solution (but I'm not sure about this).

[faysou]

Also I've noticed that in this document https://github.com/microsoft/mindaro/tree/master/samples/todo-app

It says "You will be prompted to allow the EndpointManager to run elevated and modify your hosts file."

But I didn't get any prompt. I guess this is related to the "Failed to get elevation requests" error.

[amsoedal]

@faysou, thanks for sending your logs over. It looks like there's an SSL issue when we try to communicate with the cluster. Are you currently able to make calls to the kube-api using your kubeconfig? If so, is the kube-api server locked down a certain way using SSL? Also, what cluster provider are you using?

[faysou]

Thank you for your reply. I'm using a local Openshift CRC cluster in Windows. Usually I use the integrated oc cli from CRC. I'll try tomorrow what you suggest with kubeconfig.

[faysou]

I'm able to access the cluster with kubeconfig. I used this example in python : https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/

Maybe the dotnet code using by the Bridge to kubernetes extension expects some SSL security that is not present in a local dev cluster ? Although your documentation mentions that the extension works with minikube.

It's hard to know for me without being able to debug the extension.

[amsoedal]

@faysou thanks for the quick reply! Sorry if you've already done this, but could you run a command from kubectl directly, e.g. kubectl get ns? I want to confirm if there's anything different about accessing the cluster programatically vs. from the shell directly. Thanks for providing these details.

[faysou]

The first time I got this

kubectl get ns error: You must be logged in to the server (Unauthorized)

then I did oc login -u kubeadmin -p D4tTL-hJYdy-qe36Y-okTwh https://api.crc.testing:6443 Login successful.

You have access to 62 projects, the list has been suppressed. You can list all projects with ' projects'

Using project "pol".

and this time it worked kubectl get ns NAME STATUS AGE default Active 15d gotest Active 2d21h gotest2 Active 2d20h kube-node-lease Active 15d kube-public Active 15d kube-system Active 15d openshift Active 15d openshift-apiserver Active 15d openshift-apiserver-operator Active 15d openshift-authentication Active 15d openshift-authentication-operator Active 15d openshift-cloud-credential-operator Active 15d openshift-cluster-csi-drivers Active 15d openshift-cluster-machine-approver Active 15d openshift-cluster-node-tuning-operator Active 15d openshift-cluster-samples-operator Active 15d openshift-cluster-storage-operator Active 15d openshift-cluster-version Active 15d openshift-config Active 15d openshift-config-managed Active 15d openshift-config-operator Active 15d openshift-console Active 15d openshift-console-operator Active 15d openshift-controller-manager Active 15d openshift-controller-manager-operator Active 15d openshift-dns Active 15d openshift-dns-operator Active 15d openshift-etcd Active 15d openshift-etcd-operator Active 15d openshift-image-registry Active 15d openshift-infra Active 15d openshift-ingress Active 15d openshift-ingress-operator Active 15d openshift-insights Active 15d openshift-kni-infra Active 15d openshift-kube-apiserver Active 15d openshift-kube-apiserver-operator Active 15d openshift-kube-controller-manager Active 15d openshift-kube-controller-manager-operator Active 15d openshift-kube-scheduler Active 15d openshift-kube-scheduler-operator Active 15d openshift-kube-storage-version-migrator Active 15d openshift-kube-storage-version-migrator-operator Active 15d openshift-machine-api Active 15d openshift-machine-config-operator Active 15d openshift-marketplace Active 15d openshift-monitoring Active 15d openshift-multus Active 15d openshift-network-operator Active 15d openshift-node Active 15d openshift-oauth-apiserver Active 15d openshift-openstack-infra Active 15d openshift-operator-lifecycle-manager Active 15d openshift-operators Active 15d openshift-ovirt-infra Active 15d openshift-sdn Active 15d openshift-service-ca Active 15d openshift-service-ca-operator Active 15d openshift-user-workload-monitoring Active 15d openshift-vsphere-infra Active 15d pol Active 42h squash-debugger Active 3d3h

So maybe some login must be provided in the plugin ? (just a guess, I have no idea)

Thank you for your help

You could reproduce my case by using a Codeready Container (CRC) Openshift local cluster, if you're interested (so you avoid losing time the crc program must be executed from drive C:). https://developers.redhat.com/products/codeready-containers/overview

[amsoedal]

@faysou thanks for giving that a try! Again, this might be a naive question, but what happens if you run Bridge now that you've run oc login on the command line? I had a look at some openshift docs and it seems like once you run oc login, it saves the configuration you selected.

[faysou]

I get exactly the same error.

There is some connection somehow from the plugin to the cluster as I'm able (and was able before as well) to run the configure step where I say which pod I want to replace, this step works and generates the following launch.json file, it's when it tries to do something with the cluster that it doesn't work.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch Package with Kubernetes",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}",
            "preLaunchTask": "bridge-to-kubernetes.service",
            "env": {
                "GRPC_DNS_RESOLVER": "native"
            }
        },
        {
            "name": "Launch Package",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}"
        }
    ]
}

[amsoedal]

@faysou I've logged a bug for us to investigate. It's interesting that the configuration step works fine, but not the prep-connect. It might be one of our dependencies that is causing an issue.

[faysou]

Thank you, I'm looking forward that your magical extension works in this setting. I haven't tested if it works in a real openshift cluster as I don't have access to one. But microsoft is big on Openshift with ARO.

Is the source of this extension open source ? I haven't found where is the code if it's available.

[amsoedal]

Hey @faysou, thanks again for your comments. We're working through how we can work with OpenShift right now and we'll hopefully have some updates by the end of the month. I did end up trying to create a CRC cluster locally, and I had to make this change to my kubeconfig:

- cluster: insecure-skip-tls-verify: true server: https://api.crc.testing:6443

Let me know if you try again & if it works for you!

[faysou]

Thank you @amsoedal

I've modified the config file in the .kube file and it goes indeed further in the process, the endpoint manager starts, but it doesn't work reliably.

I still get SSL errors after though, it worked further once but not after, so it's not a stable fix yet.

1) I've used this project as simple test code in Openshift, adding a component "+Add">"From Git" in the developer console: https://github.com/sclorg/golang-ex

2) Another thing I tried, when endpoint managed to work, I was using an odo "source to code" pipeline (https://odo.dev, this can push the code from the local directory when the instructions below are executed in the root directory of golang-ex, and build the code) odo create golang:latest --s2i --port 8080 odo url create odo push

And I got an error that ConfigChange was not supported because a custom command was executed (/opt/odo/bin/go-init)

So more food for thought for you and you have basic examples above to try to reproduce what I saw.

Thank you very much for your help.