Unable to debug gRPC Service that is called over SSL

[andywahr]

• Windows 10 20H2
• minikube version: v1.18.1 (hosted in Hyper-V) (also tried Docker hosted Kubernetes and minikube provisioned thru Docker)
• Backend Container (access via gRPC over HTTPS, ssl certificate is self-signed) - .NET 5 - using port 11444 for SSL
• Web API Container (REST) - ignores Certificate errors when making gRPC calls - .NET 5
• Attempted to debug Backend Container with and without routing isolation

I'm trying to debug the Backend Container. I issue my test request without "Bridge to Kubernetes" running, I successfully call the service. When I run the Backend Container thru Bridge to Kubernetes, the client reports an immediate error: "HttpRequestException: The SSL connection could not be established, see inner exception. IOException: Unable to read data from the transport connection: Connection reset by peer. SocketException: Connection reset by peer"

To Reproduce

• Steps to follow to reproduce this issue:
• minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"
• kubectl config set-context minikube
• Push 2 containers, one that calls the other via gRPC using SSL
  • To use a self-signed certificate, configure the gRPC client in Web Site new HttpClientHandler() { ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator, }
• Verify the calls works
• Debug the Backend Container via Bridge to Kubernetes
• Repeat call and verify Web is unable to connect to service

I can provide snippets from the logs, would prefer to not just upload them all. Going thru them I did see 2 WARNGs:

• 2021-04-09T03:50:01.0045262Z | Library | WARNG | Watching 'NAMESPACE/USER-PODNAME' failed, error Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request...
• 2021-04-09T03:50:01.0051940Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-WaitForWorkloadStopped {"properties":{"result":"Failed"},"metrics":{"durationInMs":17512.0}}
• 2021-04-09T03:50:01.0062005Z | Library | TRACE | Event: ConnectManagementClient-WaitRemoteAgentChange {"properties":{"result":"Failed"},"metrics":{"durationInMs":17515.0}} 2- 021-04-09T03:50:01.0209702Z | Library | WARNG | Logging handled exception: System.Threading.Tasks.TaskCanceledException: {"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Threading.Tasks.TaskExtensions.WhenAnyWithErrorPropagation[T](IEnumerable1 tasks)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.WaitForWorkloadStoppedAsync(CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass17_0.<<WaitRemoteAgentChangeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233029,"Source":"Microsoft.BridgeToKubernetes.Common","WatsonBuckets":null}

[amsoedal]

Hey @andywahr, thanks so much for providing this info. If you feel more comfortable, could you please send us a zip of your log directory at BridgeToKubernetes@microsoft.com? Having the full logs will be helpful!

[andywahr]

Here are the logs, I stopped the EndpointManager process manually to flush the logs after the error. bridge-mindarocli-2021-04-09-12-41-15-45456.zip

[andywahr]

Sorry also forgot to mention:

• Visual Studio 2019 Version 16.9.2
• Bridge to Kubernetes 0.1

[mmikkola]

I'm also experiencing this with VS 2019 16.10.3 and Bridge to Kubernetes 0.1.

• Local Kubernetes cluster, Flannel networking
• Frontend container exposed via NodePort (A gRPC endpoint, HTTPS on port 5000, .NET 5)
• Have not tried routing isolation

When the debug session is live the connection to the nodeport simply gets reset. Connecting to localhost:5000 on the machine running VS2019 works.

I've also tried running a backend service, and getting the same result (connection reset)