microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

An unexpected error occurred: 'Object reference not set to an instance of an object.' #288

Open stevengibbszions opened 2 years ago

stevengibbszions commented 2 years ago

Describe the bug Bridge to Kubernetes FAILS to start and perform redirection.

To Reproduce

  1. Launch Bridge to Kubernetes task for a service running in my k8s cluster
  2. Observe error in logs
    
    > Executing task: bridge-to-kubernetes.resource <

Redirecting Kubernetes service my-service to your machine... Target cluster: my-dvlp Current cluster: my-dvlp Target namespace: mercury Current namespace: mercury Target service name: my-service Target service ports: 8080 Using kubernetes service environment variables: true

Retrieving the current context and credentials... Validating the credentials to access the cluster... Validating the requirements to replicate resources locally... Redirecting traffic from the cluster to your machine... Connect operation failed. Stopping workload and cleaning up... An unexpected error occurred: 'Object reference not set to an instance of an object.' To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report. For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.

Error: Failed to establish a connection. Error: Connect operation failed. An unexpected error occurred: 'Object reference not set to an instance of an object.' To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report. For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.

The terminal process terminated with exit code: 1.

**Bridge Library Logs**
```logs
2022-02-24T01:43:12.1069959Z | Library | TRACE | Pulling kubeconfig...\nOperation context: <json>{"clientRequestId":null,"correlationId":"d952c87d-7f0a-44fc-82ae-63cd35b2950f1645666937354:893c9bb31c1c:41db16b6ac9a","requestId":null,"userSubscriptionId":null,"startTime":"2022-02-24T01:43:12.0470190Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220125.5","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":19428,"targetEnvironment":"Production"}}</json>
2022-02-24T01:43:12.4662187Z | Library | TRACE | Event: CloudProvider <json>{"properties":{"clusterFQDNDomain":"zionsbank.com"},"metrics":null}</json>
2022-02-24T01:43:14.2522213Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:14.2657180Z | Library | TRACE | Event: KubernetesManagementClient-CheckCredentialsAsync <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":513.0}}</json>\nOperation context: <json>{"clientRequestId":"401c90bc-86a0-4888-8cf2-057da3d8ddad","correlationId":"d952c87d-7f0a-44fc-82ae-63cd35b2950f1645666937354:893c9bb31c1c:6998ee201b6d","requestId":null,"userSubscriptionId":null,"startTime":"2022-02-24T01:43:12.0470190Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220125.5","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":19428,"targetEnvironment":"Production"}}</json>
2022-02-24T01:43:14.8379185Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1ServiceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:14.9569746Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:14.9584660Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-GetPodsFromService <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":117.0}}</json>\nOperation context: <json>{"clientRequestId":null,"correlationId":"d952c87d-7f0a-44fc-82ae-63cd35b2950f1645666937354:893c9bb31c1c:dc862d76110f","requestId":null,"userSubscriptionId":null,"startTime":"2022-02-24T01:43:12.0470190Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220125.5","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":19428,"targetEnvironment":"Production","isRoutingEnabled":false}}</json>
2022-02-24T01:43:14.9587477Z | Library | TRACE | Resolved 1 from service.
2022-02-24T01:43:14.9595981Z | Library | TRACE | Max number of containers in pod: 1, min number of containers in pod: 1
2022-02-24T01:43:14.9599588Z | Library | TRACE | Chose pod 'my-service-deployment-6d9697c545-8klnh' with 1 containers
2022-02-24T01:43:14.9647531Z | Library | TRACE | Pod my-service-deployment-6d9697c545-8klnh owned by service my-service contains 1 containers.
2022-02-24T01:43:14.9652273Z | Library | TRACE | Resolved source container my-service
2022-02-24T01:43:14.9654919Z | Library | TRACE | Resolved 1 containers with exposed ports.
2022-02-24T01:43:14.9655999Z | Library | TRACE | Successfully got container from service and pod.
2022-02-24T01:43:14.9718893Z | Library | TRACE | Owning object type: ReplicaSet
2022-02-24T01:43:15.0318362Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1ReplicaSetAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:15.0921617Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:15.0925579Z | Library | TRACE | Resolved backing deployment: my-service-deployment
2022-02-24T01:43:15.1229212Z | Library | TRACE | Invoking kubectl GetContainerEnvironment command: exec my-service-deployment-6d9697c545-8klnh -c my-service -n mercury -- env
2022-02-24T01:43:15.1242125Z | Library | TRACE | Using kubectl found at: '/home/vscode/.vscode-server/data/User/globalStorage/mindaro.mindaro/file-downloader-downloads/binaries/kubectl/linux/kubectl'
2022-02-24T01:43:16.0195792Z | Library | TRACE | Invoked kubectl GetContainerEnvironment command: 'exec my-service-deployment-6d9697c545-8klnh -c my-service -n mercury -- env' exited with 0 in 891ms
2022-02-24T01:43:16.0981124Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListServicesInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.1067724Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.1682574Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.1695845Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.2003803Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.2007089Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.2315570Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.2318137Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.2591288Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.2594045Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.2874521Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.2877902Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.3227159Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.3243273Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.3573072Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.3582587Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-02-24T01:43:16.4081599Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-02-24T01:43:16.4123653Z | Library | TRACE | Event: WorkloadInformationProvider-GetReachableEndpoints <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":387.0}}</json>
2022-02-24T01:43:16.4152861Z | Library | TRACE | Event: ConnectManagementClient-GetContainerWorkloadInfo <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":1319.0}}</json>
2022-02-24T01:43:16.4159857Z | Library | ERROR | Failed to get workload info.
2022-02-24T01:43:16.4385551Z | Library | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass16_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__17_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-02-24T01:43:16.4484644Z | Library | TRACE | Event: ConnectManagementClient-StartRemoteAgent <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":1728.0}}</json>
2022-02-24T01:43:16.4490967Z | Library | ERROR | Failed to start remote agent.
2022-02-24T01:43:16.4525874Z | Library | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass16_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__17_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass20_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-02-24T01:43:16.4702468Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-Restore <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":0.0}}</json>
2022-02-24T01:43:16.4705392Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-RestoreRemoteAgent <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":8.0}}</json>
2022-02-24T01:43:16.4706502Z | Library | TRACE | Event: ConnectManagementClient-RestoreOriginalRemoteContainer <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":9.0}}</json>
2022-02-24T01:43:16.4728414Z | Library | TRACE | Event: LocalEnvironmentManager-StopWorkload <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":0.0}}</json>
2022-02-24T01:43:16.4730144Z | Library | TRACE | Event: ConnectManagementClient-StopLocalConnection <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":1.0}}</json>

Bridge Mindaro CLI Logs

2022-02-24T01:43:12.0016055Z | MindaroCli | TRACE | Event: Command.Start <json>{"properties":{"arguments":"connect --service my-service --env /tmp/tmp-18077cfalxgzrwpbn.env --script /tmp/tmp-18077cfalxgzrwpbn.env.cmd --control-port 55414 --ppid 17828 --namespace mercury --use-kubernetes-service-environment-variables --local-port 8080","isRoutingEnabled":"false"},"metrics":null}</json>\nOperation context: <json>{"clientRequestId":null,"correlationId":"d952c87d-7f0a-44fc-82ae-63cd35b2950f1645666937354:893c9bb31c1c","requestId":null,"userSubscriptionId":null,"startTime":"2022-02-24T01:43:11.7630196Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220125.5","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":19428,"targetEnvironment":"Production","commandId":"893c9bb31c1c"}}</json>
2022-02-24T01:43:12.0749922Z | MindaroCli | TRACE | Running Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand...\nOperation context: <json>{"clientRequestId":null,"correlationId":"d952c87d-7f0a-44fc-82ae-63cd35b2950f1645666937354:893c9bb31c1c","requestId":null,"userSubscriptionId":null,"startTime":"2022-02-24T01:43:11.7630196Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220125.5","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.9","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":19428,"targetEnvironment":"Production","commandId":"893c9bb31c1c","targetServiceName":"my-service","isRoutingEnabled":false}}</json>
2022-02-24T01:43:14.6708163Z | MindaroCli | TRACE | Remoting started listening on 55414
2022-02-24T01:43:16.4552630Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>
2022-02-24T01:43:16.4569540Z | MindaroCli | ERROR | ServiceConnectCommand.ExecuteInnerAsync caught exception System.NullReferenceException: Object reference not set to an instance of an object.\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass16_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__17_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass20_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.StartRemoteAgentAsync(CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)
2022-02-24T01:43:16.4588159Z | MindaroCli | ERROR | Connect operation failed.\n
2022-02-24T01:43:16.4601450Z | MindaroCli | TRACE | Stopping workload and cleaning up...\n
2022-02-24T01:43:16.4736087Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>
2022-02-24T01:43:16.8839870Z | MindaroCli | ERROR | An unexpected error occurred: 'Object reference not set to an instance of an object.'\n
2022-02-24T01:43:16.8843111Z | MindaroCli | ERROR | To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.\n
2022-02-24T01:43:16.8847205Z | MindaroCli | ERROR | For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.\n
2022-02-24T01:43:16.8891478Z | MindaroCli | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass16_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__17_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass20_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.StartRemoteAgentAsync(CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteAsync()\n   at Microsoft.BridgeToKubernetes.Exe.CliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.CliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-02-24T01:43:16.8911653Z | MindaroCli | TRACE | Event: Command.End <json>{"properties":{"arguments":"connect --service my-service --env /tmp/tmp-18077cfalxgzrwpbn.env --script /tmp/tmp-18077cfalxgzrwpbn.env.cmd --control-port 55414 --ppid 17828 --namespace mercury --use-kubernetes-service-environment-variables --local-port 8080","result":"Failed","failureReason":"Object reference not set to an instance of an object."},"metrics":{"duration":4963.0}}</json>

Environment Details Client used: VS Code -- using Remote SSH Client's version: mindaro.mindaro@1.0.120220125 Operating System: linux

Additional context Service Spec:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: my-release
    meta.helm.sh/release-namespace: mercury
  creationTimestamp: "2022-02-19T00:15:42Z"
  labels:
    app: my-service
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":8080,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: Go-http-client
    operation: Update
    time: "2022-02-19T00:15:42Z"
  name: my-service
  namespace: mercury
  resourceVersion: "640046229"
  uid: dc12fcd4-3894-46cb-9d7e-ee9b90f1a5a4
spec:
  clusterIP: 10.43.183.61
  clusterIPs:
  - 10.43.183.61
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: my-service
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

tasks.json

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "bridge-to-kubernetes.resource",
      "type": "bridge-to-kubernetes.resource",
      "resource": "my-service",
      "resourceType": "service",
      "ports": [
        8080
      ],
      "targetCluster": "my-dvlp",
      "targetNamespace": "mercury",
      "useKubernetesServiceEnvironmentVariables": true
    }
  ]
}
amsoedal commented 2 years ago

Hi @stevengibbszions, I've produced a new CLI build that contains extra logging information as well as a tiny code change. Can you test it out and 1) see if anything changes, 2) send us the new detailed logs?

Instructions:

  1. Download the zipped CLI for your platform here: Linux, Windows, OSX
  2. Extract the zipped folder
  3. Before launching VS Code, set these env vars: BRIDGE_BUILD_PATH=<the directory created by step 2>, BRIDGE_BINARYUTILITYVERSION="v2"
  4. Launch VS Code such that the environment variables are set (e.g. code . from inside the shell where you set the env vars) and run Bridge as usual
stevengibbszions commented 2 years ago

Hello @amsoedal and thanks for your reply!

Here are the steps I took in order to get your code to run.

  1. Downloaded and extracted lpk-linux zip file
  2. Rename "dsc" binary to "bridge"
  3. chmod +x on kubectl, bridge and EventManager
  4. Set ENV variables as suggested.

Here are the results:

FAIL -- Object reference not set to an instance of an object. Logs - from Terminal

> Executing task: bridge-to-kubernetes.resource <

Redirecting Kubernetes service my-service to your machine...
Target cluster: my-dvlp
Current cluster: my-dvlp
Target namespace: mercury
Current namespace: mercury
Target service name: my-service
Target service ports: 8080
Using kubernetes service environment variables: true

Retrieving the current context and credentials...
Validating the credentials to access the cluster...
Validating the requirements to replicate resources locally...
Redirecting traffic from the cluster to your machine...
Connect operation failed.
Stopping workload and cleaning up...
An unexpected error occurred: 'Object reference not set to an instance of an object.'
To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.
For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.

Error: Failed to establish a connection. Error: Connect operation failed.
An unexpected error occurred: 'Object reference not set to an instance of an object.'
To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.
For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.

The terminal process terminated with exit code: 1.

Logs - bridge-mindarocli

2022-03-05T20:44:20.8730779Z | MindaroCli | TRACE | Event: Command.Start <json>{"properties":{"arguments":"connect --service my-service --env /tmp/tmp-20570gjc3yxqx9mfm.env --script /tmp/tmp-20570gjc3yxqx9mfm.env.cmd --control-port 54520 --ppid 20232 --namespace mercury --use-kubernetes-service-environment-variables --local-port 8080","isRoutingEnabled":"false"},"metrics":null}</json>\nOperation context: <json>{"clientRequestId":null,"correlationId":"f9f408b0-5d3c-49f9-8dbc-1642d80468761646511228998:b7f82cfd353d","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-05T20:44:20.6256896Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220224.1","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.6","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":29679,"targetEnvironment":"Production","commandId":"b7f82cfd353d"}}</json>
2022-03-05T20:44:20.9502323Z | MindaroCli | TRACE | Running Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand...\nOperation context: <json>{"clientRequestId":null,"correlationId":"f9f408b0-5d3c-49f9-8dbc-1642d80468761646511228998:b7f82cfd353d","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-05T20:44:20.6256896Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220224.1","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.6","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":29679,"targetEnvironment":"Production","commandId":"b7f82cfd353d","targetServiceName":"my-service","isRoutingEnabled":false}}</json>
2022-03-05T20:44:22.8026548Z | MindaroCli | TRACE | Remoting started listening on 54520
2022-03-05T20:44:24.5948368Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>
2022-03-05T20:44:24.5962222Z | MindaroCli | ERROR | ServiceConnectCommand.ExecuteInnerAsync caught exception System.NullReferenceException: Object reference not set to an instance of an object.\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass17_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__18_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass21_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.StartRemoteAgentAsync(CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)
2022-03-05T20:44:24.5977725Z | MindaroCli | ERROR | Connect operation failed.\n
2022-03-05T20:44:24.5988872Z | MindaroCli | TRACE | Stopping workload and cleaning up...\n
2022-03-05T20:44:24.6134055Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>
2022-03-05T20:44:24.6902227Z | MindaroCli | ERROR | An unexpected error occurred: 'Object reference not set to an instance of an object.'\n
2022-03-05T20:44:24.6905536Z | MindaroCli | ERROR | To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.\n
2022-03-05T20:44:24.6911370Z | MindaroCli | ERROR | For diagnostic information, see logs at '/tmp/Bridge To Kubernetes'.\n
2022-03-05T20:44:24.6949117Z | MindaroCli | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass17_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__18_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass21_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.StartRemoteAgentAsync(CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteInnerAsync(IConnectManagementClient connectManagementClient, Action workloadStartedHandler, CancellationToken cancellationToken, IRoutingManagementClient routingManagementClient)\n   at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.ConnectCommand.ExecuteAsync()\n   at Microsoft.BridgeToKubernetes.Exe.CliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Exe.CliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-03-05T20:44:24.6964807Z | MindaroCli | TRACE | Event: Command.End <json>{"properties":{"arguments":"connect --service my-service --env /tmp/tmp-20570gjc3yxqx9mfm.env --script /tmp/tmp-20570gjc3yxqx9mfm.env.cmd --control-port 54520 --ppid 20232 --namespace mercury --use-kubernetes-service-environment-variables --local-port 8080","result":"Failed","failureReason":"Object reference not set to an instance of an object."},"metrics":{"duration":3901.0}}</json>

Logs -- Bridge Library

2022-03-05T20:44:20.9917096Z | Library | TRACE | Pulling kubeconfig...\nOperation context: <json>{"clientRequestId":null,"correlationId":"f9f408b0-5d3c-49f9-8dbc-1642d80468761646511228998:b7f82cfd353d:c91e65fa5b57","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-05T20:44:20.9200673Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220224.1","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.6","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":29679,"targetEnvironment":"Production"}}</json>
2022-03-05T20:44:21.3391411Z | Library | TRACE | Event: CloudProvider <json>{"properties":{"clusterFQDNDomain":"zionsbank.com"},"metrics":null}</json>
2022-03-05T20:44:22.2716368Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:22.2847650Z | Library | TRACE | Event: KubernetesManagementClient-CheckCredentialsAsync <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":479.0}}</json>\nOperation context: <json>{"clientRequestId":"f3051e53-6867-4578-b3c7-5d327395d095","correlationId":"f9f408b0-5d3c-49f9-8dbc-1642d80468761646511228998:b7f82cfd353d:ca8ebc9f6925","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-05T20:44:20.9200673Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220224.1","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.6","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":29679,"targetEnvironment":"Production"}}</json>
2022-03-05T20:44:22.9665241Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1ServiceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:23.0804981Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:23.0820680Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-GetPodsFromService <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":112.0}}</json>\nOperation context: <json>{"clientRequestId":null,"correlationId":"f9f408b0-5d3c-49f9-8dbc-1642d80468761646511228998:b7f82cfd353d:b38fdfe92b27","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-05T20:44:20.9200673Z","userAgent":"VSCode/1.0.120220125","requestHttpMethod":null,"requestUri":null,"version":"1.0.20220224.1","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021","framework":".NET Core 3.1.6","macAddressHash":"dcc5aa70fd400e263d3ed43e9a8d884cdf23d8f27220f2d6f6083758d5409009","processId":29679,"targetEnvironment":"Production","isRoutingEnabled":false}}</json>
2022-03-05T20:44:23.0823314Z | Library | TRACE | Resolved 1 from service.
2022-03-05T20:44:23.0830346Z | Library | TRACE | Max number of containers in pod: 1, min number of containers in pod: 1
2022-03-05T20:44:23.0834160Z | Library | TRACE | Chose pod 'my-service-deployment-7dd85d8669-wvb59' with 1 containers
2022-03-05T20:44:23.0881708Z | Library | TRACE | Pod my-service-deployment-7dd85d8669-wvb59 owned by service my-service contains 1 containers.
2022-03-05T20:44:23.0886943Z | Library | TRACE | Resolved source container my-service
2022-03-05T20:44:23.0889318Z | Library | TRACE | Resolved 1 containers with exposed ports.
2022-03-05T20:44:23.0890628Z | Library | TRACE | Successfully got container from service and pod.
2022-03-05T20:44:23.0945328Z | Library | TRACE | Owning object type: ReplicaSet
2022-03-05T20:44:23.1690705Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1ReplicaSetAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:23.2338264Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:23.2342318Z | Library | TRACE | Resolved backing deployment: my-service-deployment
2022-03-05T20:44:23.2704276Z | Library | TRACE | Invoking kubectl GetContainerEnvironment command: exec my-service-deployment-7dd85d8669-wvb59 -c my-service -n mercury -- env
2022-03-05T20:44:23.2718459Z | Library | TRACE | Using kubectl found at: '/app/bridge-to-k8s-lpk-linux/kubectl/linux/kubectl'
2022-03-05T20:44:23.9387368Z | Library | TRACE | Invoked kubectl GetContainerEnvironment command: 'exec my-service-deployment-7dd85d8669-wvb59 -c my-service -n mercury -- env' exited with 0 in 663ms
2022-03-05T20:44:24.0488272Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"ListServicesInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.0578614Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.1195988Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.1208266Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.1538128Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.1541238Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.1896287Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.1899336Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.2240892Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.2244425Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.3005035Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.3025650Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.3554581Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.3564426Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.4382154Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.4389453Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.4746987Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.4749947Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.5073361Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.5075795Z | Library | TRACE | Detected headless service. Looking for endpoints with same name as service...
2022-03-05T20:44:24.5434065Z | Library | TRACE | Dependency: Kubernetes <json>{"target":"GetEndpointInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2022-03-05T20:44:24.5478301Z | Library | TRACE | Event: WorkloadInformationProvider-GetReachableEndpoints <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":602.0}}</json>
2022-03-05T20:44:24.5491339Z | Library | TRACE | Event: WorkloadInformationProvider-GatherWorkloadInfo <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":1296.0}}</json>
2022-03-05T20:44:24.5521874Z | Library | ERROR | Failed to get workload info.
2022-03-05T20:44:24.5804105Z | Library | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass17_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__18_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-03-05T20:44:24.5892292Z | Library | TRACE | Event: ConnectManagementClient-StartRemoteAgent <json>{"properties":{"result":"Failed"},"metrics":{"durationInMs":1720.0}}</json>
2022-03-05T20:44:24.5896573Z | Library | ERROR | Failed to start remote agent.
2022-03-05T20:44:24.5925514Z | Library | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider._CollectServicesToRouteAsync(String workloadNamespace, IEnumerable`1 services, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GetReachableEndpointsAsync(String namespaceName, ILocalProcessConfig localProcessConfig, Boolean includeSameNamespaceServices, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.Connect.WorkloadInformationProvider.GatherWorkloadInfo(ILocalProcessConfig localProcessConfig, CancellationToken cancellationToken)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass17_0.<<-ctor>b__2>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.AsyncLazy`1.<>c__DisplayClass6_0.<<RetryOnFailure>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<GetWorkloadInfo>b__18_0()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetWorkloadInfo()\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass21_0.<<StartRemoteAgentAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Library","WatsonBuckets":null}
2022-03-05T20:44:24.6103417Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-Restore <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":0.0}}</json>
2022-03-05T20:44:24.6105931Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-RestoreRemoteAgent <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":8.0}}</json>
2022-03-05T20:44:24.6107072Z | Library | TRACE | Event: ConnectManagementClient-RestoreOriginalRemoteContainer <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":9.0}}</json>
2022-03-05T20:44:24.6126858Z | Library | TRACE | Event: LocalEnvironmentManager-StopWorkload <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":0.0}}</json>
2022-03-05T20:44:24.6128434Z | Library | TRACE | Event: ConnectManagementClient-StopLocalConnection <json>{"properties":{"result":"Succeeded"},"metrics":{"durationInMs":1.0}}</json>

Seems like its parsing through the env of the container it found. What is it looking for that it cannot find?

amsoedal commented 2 years ago

Hi @stevengibbszions, thanks for trying this out. Someone actually ran into a similar issue and it seems to be due to certain endpoints not having a targetIp. As a workaround, can you make sure this field is set for the endpoints? https://github.com/microsoft/mindaro/issues/284#issuecomment-1061197046

stevengibbszions commented 2 years ago

Hi @stevengibbszions, thanks for trying this out. Someone actually ran into a similar issue and it seems to be due to certain endpoints not having a targetIp. As a workaround, can you make sure this field is set for the endpoints? #284 (comment)

@amsoedal My Service definitely has a Target IP. So it seems the presence of a headless services in the same namespace causes this issue? For me that's a difficult workaround. We have 100 services and 10 of them are headless (with no target IP). What is the roadmap for fixing this bug?

stevengibbszions commented 2 years ago

@amsoedal Any chance this bug is getting looked at?