search

microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

Pod hosting Remote Agent doesn't become Ready #303

Open vadiml77 opened 2 years ago

vadiml77 commented 2 years ago

Describe the bug

The pod with the remote agent does get deployed, however the pod is not Ready from the K8S perspective. Consequently the service being debugged never switches to the pod hosting the agent, and continues to route to the original pod

Logs

[~] $ k logs ssp-dev-ssp-auth-mgr-67c675b7cb-w6sfc 2022-03-13T15:48:24.8934845Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}} 2022-03-13T15:48:25.5501874Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}} 2022-03-13T15:48:26.0771109Z | RemoteAgent | TRACE | SyncService.RunTar tar succeeded. \nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}} 2022-03-13T15:48:27.5501079Z | RemoteAgent | TRACE | ReversePortForwardConnector created for port 8086\nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}} 2022-03-13T15:48:27.5556324Z | RemoteAgent | TRACE | ReversePortForwardConnector start listening on port 8086 2022-03-13T15:48:41.7832360Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 8086 accepted incoming request as stream 1. 2022-03-13T15:48:41.7928172Z | RemoteAgent | TRACE | AgentHub connnected for 8086, id 1 2022-03-13T15:48:41.7940706Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 1 returns 237 bytes. 2022-03-13T15:48:41.7988469Z | RemoteAgent | TRACE | AgentHub received for 8086, id 1, size 237 2022-03-13T15:48:41.9526498Z | RemoteAgent | TRACE | AgentHub disconnect for 8086, id 1\nOperation context: {"clientRequestId":"721219b1-ac86-4e2e-a3ab-966360fb219e","correlationId":"547604da-2483-4997-b66e-767db40684b71647186379555:5c65c88fef4b:d568409433ec","requestId":null,"userSubscriptionId":null,"startTime":"2022-03-13T15:48:17.5916105Z","userAgent":"RemoteAgent/1.0.0.0","requestHttpMethod":null,"requestUri":null,"version":"1.0.0.0","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020","framework":".NET Core 3.1.15","processId":1,"targetEnvironment":"Production"}} 2022-03-13T15:48:41.9552076Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 1 2022-03-13T15:48:41.9566303Z | RemoteAgent | TRACE | AgentHub closed for 8086, id 1 2022-03-13T15:48:41.9837605Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync exception 'Unable to read data from the transport connection: Operation canceled.' when invoking handler. Close.

Describe Pod logs for the pod with remote agent

Events: Type Reason Age From Message

Normal Scheduled Successfully assigned dev/ssp-dev-ssp-auth-mgr-67c675b7cb-w6sfc to HOSTNAME_REDACTED Normal Pulling 14m kubelet, HOSTNAME_REDACTED Pulling image "lolodi/devhostagent:20210909.1" Normal Pulled 14m kubelet, HOSTNAME_REDACTED Successfully pulled image "lolodi/devhostagent:20210909.1" in 934.946905ms Normal Created 14m kubelet, HOSTNAME_REDACTED Created container ssp-auth-mgr Normal Started 14m kubelet, HOSTNAME_REDACTED Started container ssp-auth-mgr Warning Unhealthy 11m (x3 over 13m) kubelet, HOSTNAME_REDACTED Liveness probe failed: Get "https://10.39.0.11:8086/auth/v1/health/liveness": EOF Warning Unhealthy 4m41s (x55 over 14m) kubelet, HOSTNAME_REDACTED Readiness probe failed: Get "https://10.39.0.11:8086/auth/v1/health/readiness": EOF