search

microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

Azure Application Gateway with Bridge to Kubernetes #221

Open markgarrigan opened 3 years ago

markgarrigan commented 3 years ago

Describe the bug Using Azure Application Gateway with bridge to kubernetes breaks App Gateway routing.

To Reproduce Set up Application Gateway for your AKS cluster. Use path based routing to point a url to a backend service. Connect VS Code to the backend service through bridge to kubernetes. Try to access the public url of the service. 502 HTTP Error will be returned.

Stop debugging with bridge.

After the pod is restored the routing works and returns 200 when accessing the public url.

Expected behavior Application gateway can still route to the backend service as usual after connecting with bridge.

markgarrigan commented 3 years ago

Wait. Does Bridge to Kubernetes restart the pod and run the app on port 9229? If that's the case, then the Application Gateway would break since it's sending requests to port 3000 which is the port this node app typically starts up on.

markgarrigan commented 3 years ago

Fixed. Needed to update my launch.json to this...

...
    {
      "type": "node",
      "request": "launch",
      "name": "Launch via NPM with Kubernetes",
      "runtimeExecutable": "npm",
      "runtimeArgs": [
        "run-script",
        "debug"
      ],
      "port": 3000,
      "preLaunchTask": "bridge-to-kubernetes.resource",
      "env": {
        "GRPC_DNS_RESOLVER": "native"
      }
    }
...
markgarrigan commented 3 years ago

Ok... after further review it sort of works...

It works for a while then stops. Then the pod's logs get filled with some pretty crazy logs...

2021-08-26T21:05:59.6588309Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 27 228 bytes\nOperation context: <json>{"clientRequestId":"6735f6a6-55cc-49f2-bba1-69a41c020a42","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:87b3f6ae9abf:580ca282947c","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:00:28.1343583Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:05:59.6589926Z | RemoteAgent | TRACE | Send 228 via 27
2021-08-26T21:05:59.6600946Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 27 returns 0 bytes.
2021-08-26T21:05:59.6603324Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 27.
2021-08-26T21:05:59.6605385Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 27
2021-08-26T21:05:59.6607503Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 27
2021-08-26T21:06:29.5829712Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 28.
2021-08-26T21:06:29.5834795Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 28
2021-08-26T21:06:29.5837283Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 28 returns 41 bytes.
2021-08-26T21:06:29.5839746Z | RemoteAgent | TRACE | AgentHub received for 3000, id 28, size 41
2021-08-26T21:06:29.5884169Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 29.
2021-08-26T21:06:29.5896717Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 29
2021-08-26T21:06:29.5899696Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 29 returns 41 bytes.
2021-08-26T21:06:29.5904508Z | RemoteAgent | TRACE | AgentHub received for 3000, id 29, size 41
2021-08-26T21:06:39.6788365Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 30.
2021-08-26T21:06:39.6792500Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 30
2021-08-26T21:06:39.6795582Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 30 returns 41 bytes.
2021-08-26T21:06:39.6798090Z | RemoteAgent | TRACE | AgentHub received for 3000, id 30, size 41
2021-08-26T21:06:40.3880037Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 31.
2021-08-26T21:06:40.3884680Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 31
2021-08-26T21:06:40.3887921Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 31 returns 41 bytes.
2021-08-26T21:06:40.3890765Z | RemoteAgent | TRACE | AgentHub received for 3000, id 31, size 41
2021-08-26T21:06:50.6596101Z | RemoteAgent | TRACE | ReversePortForwardConnector start listening on port 3000
2021-08-26T21:06:50.6597786Z | RemoteAgent | TRACE | ReversePortForwardConnector AcceptTcpClientAsync on port 3000 failed with Cannot access a disposed object.\nObject name: 'System.Net.Sockets.Socket'..
2021-08-26T21:06:50.6602150Z | RemoteAgent | TRACE | ReversePortForwardConnector AcceptTcpClientAsync on port 3000 failed with Cannot access a disposed object.\nObject name: 'System.Net.Sockets.Socket'..
2021-08-26T21:06:59.5812818Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 28 returns 0 bytes.
2021-08-26T21:06:59.5815295Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 28.
2021-08-26T21:06:59.5817517Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 28
2021-08-26T21:06:59.5819414Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 28
2021-08-26T21:06:59.5890186Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 29 returns 0 bytes.
2021-08-26T21:06:59.5892737Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 29.
2021-08-26T21:06:59.5894867Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 29
2021-08-26T21:06:59.5896449Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 29
2021-08-26T21:08:19.6739891Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 30 returns 0 bytes.
2021-08-26T21:08:19.6742828Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 30.
2021-08-26T21:08:19.6744264Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 30
2021-08-26T21:08:19.6745964Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 30
2021-08-26T21:08:20.3856606Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 31 returns 0 bytes.
2021-08-26T21:08:20.3858349Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 31.
2021-08-26T21:08:20.3860451Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 31
2021-08-26T21:08:20.3863349Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 31
markgarrigan commented 3 years ago

If I visit localhost:3000 that traffic is still being handled by the debugger. So the pod is still running fine and healthy. But the traffic from the Application Gateway health probe fails which makes the Application Gateway think the backend is down so it will not route any traffic from the public URL to the backend.

When I stop debugging I have it set to disconnect from kubernetes. Once that is complete, Application Gateway begins to work again.

markgarrigan commented 3 years ago

Some more logs.. It worked for maybe a minute... then died...

LAMU02Y40MKJG5J:dss mgarriga$ k logs dss-shim-
dss-shim-7f77b6787f-rwgks     dss-shim-restore-0d296-7qpzv  
LAMU02Y40MKJG5J:dss mgarriga$ k logs dss-shim-7f77b6787f-rwgks -f
2021-08-26T21:28:15.5332445Z | RemoteAgent | TRACE | ReversePortForwardConnector created for port 3000\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:28:15.5673567Z | RemoteAgent | TRACE | ReversePortForwardConnector start listening on port 3000
2021-08-26T21:28:39.8276173Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 1.
2021-08-26T21:28:39.8350711Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 1
2021-08-26T21:28:39.8362822Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 1 returns 41 bytes.
2021-08-26T21:28:39.8396139Z | RemoteAgent | TRACE | AgentHub received for 3000, id 1, size 41
2021-08-26T21:28:39.8820245Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 2.
2021-08-26T21:28:39.8824629Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 2
2021-08-26T21:28:39.8828452Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 2 returns 41 bytes.
2021-08-26T21:28:39.8831159Z | RemoteAgent | TRACE | AgentHub received for 3000, id 2, size 41
2021-08-26T21:28:39.9765217Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 1 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:28:39.9780890Z | RemoteAgent | TRACE | Send 228 via 1
2021-08-26T21:28:39.9795305Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 2 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:28:39.9798695Z | RemoteAgent | TRACE | Send 228 via 2
2021-08-26T21:28:39.9813372Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 1 returns 0 bytes.
2021-08-26T21:28:39.9818190Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 2 returns 0 bytes.
2021-08-26T21:28:39.9818720Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 1.
2021-08-26T21:28:39.9821085Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 2.
2021-08-26T21:28:39.9846736Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 1
2021-08-26T21:28:39.9847152Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 2
2021-08-26T21:28:39.9865443Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 1
2021-08-26T21:28:39.9866012Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 2
2021-08-26T21:29:09.8323599Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 3.
2021-08-26T21:29:09.8344978Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 3
2021-08-26T21:29:09.8358566Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 3 returns 41 bytes.
2021-08-26T21:29:09.8361644Z | RemoteAgent | TRACE | AgentHub received for 3000, id 3, size 41
2021-08-26T21:29:09.8804784Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 4.
2021-08-26T21:29:09.8811163Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 4
2021-08-26T21:29:09.8822351Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 4 returns 41 bytes.
2021-08-26T21:29:09.8837351Z | RemoteAgent | TRACE | AgentHub received for 3000, id 4, size 41
2021-08-26T21:29:09.9197440Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 3 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:09.9200127Z | RemoteAgent | TRACE | Send 228 via 3
2021-08-26T21:29:09.9211391Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 3 returns 0 bytes.
2021-08-26T21:29:09.9214170Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 3.
2021-08-26T21:29:09.9217000Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 3
2021-08-26T21:29:09.9219439Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 3
2021-08-26T21:29:09.9602580Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 4 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:09.9605481Z | RemoteAgent | TRACE | Send 228 via 4
2021-08-26T21:29:09.9615661Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 4 returns 0 bytes.
2021-08-26T21:29:09.9617654Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 4.
2021-08-26T21:29:09.9619668Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 4
2021-08-26T21:29:09.9622216Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 4
2021-08-26T21:29:35.5386154Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 5.
2021-08-26T21:29:35.5402053Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 5
2021-08-26T21:29:35.5417388Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 5 returns 1292 bytes.
2021-08-26T21:29:35.5426470Z | RemoteAgent | TRACE | AgentHub received for 3000, id 5, size 1292
2021-08-26T21:29:35.6136401Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 5 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:35.6148302Z | RemoteAgent | TRACE | Send 228 via 5
2021-08-26T21:29:39.8313908Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 6.
2021-08-26T21:29:39.8344313Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 6
2021-08-26T21:29:39.8366517Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 6 returns 41 bytes.
2021-08-26T21:29:39.8378719Z | RemoteAgent | TRACE | AgentHub received for 3000, id 6, size 41
2021-08-26T21:29:39.8843261Z | RemoteAgent | TRACE | ReversePortForwardConnector on port 3000 accepted incoming request as stream 7.
2021-08-26T21:29:39.8856799Z | RemoteAgent | TRACE | AgentHub connnected for 3000, id 7
2021-08-26T21:29:39.8873630Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 7 returns 41 bytes.
2021-08-26T21:29:39.8887054Z | RemoteAgent | TRACE | AgentHub received for 3000, id 7, size 41
2021-08-26T21:29:39.9338345Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 6 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:39.9346922Z | RemoteAgent | TRACE | Send 228 via 6
2021-08-26T21:29:39.9370110Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 6 returns 0 bytes.
2021-08-26T21:29:39.9372074Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 6.
2021-08-26T21:29:39.9374363Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 6
2021-08-26T21:29:39.9377018Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 6
2021-08-26T21:29:39.9835037Z | RemoteAgent | TRACE | AgentHub Send for 3000, id 7 228 bytes\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:39.9837678Z | RemoteAgent | TRACE | Send 228 via 7
2021-08-26T21:29:39.9848230Z | RemoteAgent | TRACE | ReversePortForwardConnector.HookupStreamData ReadAsync 7 returns 0 bytes.
2021-08-26T21:29:39.9851102Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync finishes stream 7.
2021-08-26T21:29:39.9852659Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 7
2021-08-26T21:29:39.9854366Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 7
2021-08-26T21:29:40.6226326Z | RemoteAgent | TRACE | AgentHub disconnect for 3000, id 5\nOperation context: <json>{"clientRequestId":"add9627d-adf8-414e-af34-a30a6e374668","correlationId":"70f97dfd-52a4-4f63-b94f-91cbf6f9e62d1629986811287:f787033d685c:9c750618f0c1","requestId":null,"userSubscriptionId":null,"startTime":"2021-08-26T21:28:09.9522301Z","userAgent":"RemoteAgent/1.0.20210819.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210819.2","requestHeaders":{},"loggingProperties":{"applicationName":"RemoteAgent","deviceOperatingSystem":"Linux 5.4.0-1051-azure #53~18.04.1-Ubuntu SMP Fri Jun 18 22:32:58 UTC 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-08-26T21:29:40.6232438Z | RemoteAgent | TRACE | PortForwardConnector.Disconnect 5
2021-08-26T21:29:40.6234754Z | RemoteAgent | TRACE | AgentHub closed for 3000, id 5
2021-08-26T21:29:40.6452061Z | RemoteAgent | TRACE | ReversePortForwardConnector.StartReceiveDataAsync exception 'Unable to read data from the transport connection: Operation canceled.' when invoking handler. Close.
daniv-msft commented 3 years ago

Thanks @markgarrigan for reopening this issue and sharing these logs! From what you describe, it looks indeed like something goes wrong with the probe. We added support for AGIC a couple of months ago, so there must be a special case we don't handle properly.

Could you please confirm that your ingress serving the health probe has the annotation azure/application-gateway? This is how we detect the AGIC scenario on our side. Also, did you configure Bridge to Kubernetes to run isolated?

markgarrigan commented 3 years ago

My ingress annotations are as follows...

kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: REDACTED
appgw.ingress.kubernetes.io/health-probe-path: "/health"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"

And BtK is not running isolated. But it can if I need it to.

daniv-msft commented 3 years ago

Thanks @markgarrigan, that helps! I synced with my team about this issue, and this might actually be independent from AGIC. We're presently investigating an other issue related to liveness probes, and the problem you encounter might just be another example of it. We're looking into it and hopefully we'll have more details to share in the coming days.

Something else that we're considering is a regression of a bug we saw in the past, where the liveness probe call is redirected to the local dev machine, but that when the customer stops on any breakpoint, this actually prevents the liveness probe from getting a successful reply, and then our remote agent is killed. From what you saw when reproducing the bug, could it be related to you putting a breakpoint in your code or the local process being unable to reply to probes?

pragyamehta commented 3 years ago

Hi @markgarrigan Thanks for all the information. Could you set the failureThreshold to a very high value and try to get the logs of the pod you are trying to debug using the command - kubectl -n logs You can send the logs to bridgetokubernetes@microsoft.com and we will investigate and get back to you.

Thanks!

markgarrigan commented 3 years ago

@pragyamehta I can do this tomorrow. Where is the failureThreshold setting? Is that a BtK setting? Thanks!

daniv-msft commented 3 years ago

@markgarrigan In this case @pragyamehta refers to the failureThreshold Kubernetes setting you can set for your liveness probe, e.g.:

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 1
  periodSeconds: 10
markgarrigan commented 3 years ago

@daniv-msft @pragyamehta Got it. I should be able to send information tomorrow. Thanks!

On Thu, Sep 2, 2021 at 9:07 PM daniv-msft @.***> wrote:

@markgarrigan https://github.com/markgarrigan In this case @pragyamehta https://github.com/pragyamehta refers to the failureThreshold Kubernetes setting you can set for your liveness probe, e.g.:

livenessProbe: httpGet: path: /healthz port: liveness-port failureThreshold: 1 periodSeconds: 10

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/mindaro/issues/221#issuecomment-912196343, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADHD5ZQOB5GIRXTEIXLUQTUAAUVNANCNFSM5C3SAZNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.