Unable to start remote agent for headless service

[henrikmilling]

Hi, We have been trying to debug a very simple headless service that only listen as a consumer on Kafka. No ports open - just a simple BackgroundService that subscribe for a Kafka topic. But no matter what we have tried (upgraded VS2019 to 16.10.4 - and two different versions of Bridge to Kubernetes) B2k8s fails to start the service with a connection failure: "Object reference not set to an instance of an object". More info from the log files:

2021-07-27T15:09:39.2961112Z | MindaroCli | TRACE | Initializing progress reporter... 2021-07-27T15:09:43.7355749Z | MindaroCli | TRACE | Remoting started listening on 58894 2021-07-27T15:09:46.7630525Z | MindaroCli | TRACE | Waiting for EndpointManager to come up ...\n 2021-07-27T15:09:49.6256178Z | MindaroCli | TRACE | EndpointManager came up successfully.\n 2021-07-27T15:09:55.4870451Z | MindaroCli | TRACE | Waiting for 'interimingress-7c964dd894-lmfzt' in namespace 'nsight-ingress' to reach running state...\n 2021-07-27T15:09:58.1157304Z | MindaroCli | TRACE | Deployment 'nsight-ingress/interimingress' patched to run agent.\n 2021-07-27T15:09:58.1172410Z | MindaroCli | TRACE | Found container 'interimingress' in pod 'interimingress-7c964dd894-lmfzt'.\n 2021-07-27T15:09:58.1211305Z | MindaroCli | TRACE | Preparing to run Bridge To Kubernetes configured as pod nsight-ingress/interimingress-7c964dd894-lmfzt ...\n 2021-07-27T15:09:59.5102749Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward {"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}} 2021-07-27T15:09:59.5115733Z | MindaroCli | ERROR | ServiceConnectCommand.ExecuteInnerAsync caught exception System.NullReferenceException: Object reference not set to an instance of an object.\n at System.Linq.Enumerable.SelectManySingleSelectorIterator2.MoveNext()\n at System.Linq.Enumerable.DistinctIterator1.MoveNext()\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager._GatherContainerWorkloadInfo(RemoteContainerConnectionDetails remoteContainerConnectionDetails, ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.StartRemoteAgentAsync(ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass15_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func1 func, FailureConfig failureConfig)\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.StartRemoteAgentAsync(IProgress1 progress, CancellationToken cancellationToken)\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action`1 workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient) 2021-07-27T15:09:59.5124609Z | MindaroCli | ERROR | Connect operation failed.\n

... and ...

2021-07-27T15:09:58.8380206Z | Library | TRACE | Port forward interimingress-7c964dd894-lmfzt 58907:50051 58910 : Run receive loop 2021-07-27T15:09:59.0949191Z | Library | TRACE | Reset completed 2021-07-27T15:09:59.2982608Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-StartRemoteAgent {"properties":{"result":"Failed"},"metrics":{"durationInMs":8702.0}} 2021-07-27T15:09:59.2985418Z | Library | TRACE | Event: ConnectManagementClient-StartRemoteAgent {"properties":{"agentHostingMode":"2","sourceEntityType":"1","namespaceName":"nsight-ingress","containerName":"interimingress","serviceName":"interimingress","deploymentName":"interimingress","podName":"interimingress-665776c7-gtl96","routingHeaderValue":"","localProcessConfigFilePath":"KubernetesLocalProcessConfig.yaml","result":"Failed"},"metrics":{"durationInMs":9084.0}} 2021-07-27T15:09:59.2990300Z | Library | ERROR | Failed to start remote agent. 2021-07-27T15:09:59.4991706Z | Library | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Linq.Enumerable.SelectManySingleSelectorIterator2.MoveNext()\r\n at System.Linq.Enumerable.DistinctIterator1.MoveNext()\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager._GatherContainerWorkloadInfo(RemoteContainerConnectionDetails remoteContainerConnectionDetails, ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.StartRemoteAgentAsync(ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass15_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"System.Linq","WatsonBuckets":null} 2021-07-27T15:09:59.5321882Z | Library | TRACE | Event: LocalEnvironmentManager-StopWorkload {"properties":{"result":"Succeeded"},"metrics":{"durationInMs":0.0}} 2021-07-27T15:09:59.5323668Z | Library | TRACE | Event: ConnectManagementClient-StopLocalConnection {"properties":{"result":"Succeeded"},"metrics":{"durationInMs":1.0}}

The only pods in the namespace is an api-service (port 80/443) and the above mentioned headless service. No problems on debugging the api-service (as well as other services) - but it seems to be a problem to debug headless services? Maybe related to issue #151?

[amsoedal]

Hi @henrikmilling, thanks for reaching out. We need to handle this null ref. I think this case gets triggered in our code when no ports are defined on any of the specs for the services in the namespace. That might by why you're reproducing it, since you mention no ports are open. We have a ticket open on our side to investigate further -- in the meantime, adding a port to the service spec will probably get around this (although I understand it's not ideal to edit your project setup).

The other thing I'll mention is that our releases make it out slower to VS than VS Code -- if you're able to make it past this issue but you run into other problems, I suggest trying VS Code since it includes the most recent fixes.

[henrikmilling]

Great! :-) Thanks a lot 👍 It works fine if we add ports to the deployment/service - and for now - this is an acceptable work around. Just some detail information (if it can be of any value to you/others): The api-service/pod already had ports open. It was only the headless service that we tried to debug that did not have any ports (These two services/pods beeing the only ones in the given namespace) . But adding 80/443 to the service being debugged solved the problem for now :-)

Thanks for responding this fast - deeply appreciated :-)