VS Code notebook loses connection to running kernel over night.

[greazer]

Expected Behavior

Connection to running notebook on an Azure ML compute remains even after many hours

Actual Behavior

Disconnect occurs at some point with a modal dialog stating "Cannot reconnect. Please reload the window". This is coming from VS Code core (or remote services).

Steps to Reproduce the Problem

1. Create a notebook with a code cell like this:

2. Let it run overnight. By morning you'll see something like this:

Per Rob on the VS Code team, this disconnection is expected. There may be ways of ensuring it doesn't happen via the Azure ML extension. This could be problematic for cost (though I'm not sure how it's different than running a kernel all night).

Barring the prevention of disconnect, VS Code core and the Jupyter extension team may need to develop a way to explicitly disconnect and reconnect to running kernels.

[sevillal]

Reconnection

Regarding the reconnection issue here is further explanation on the cause, the timestamps shown are just to depict an approximation of how events would happen:

• 10 PM Connection is open
• 11 PM
  • Local compute goes to sleep
  • Websocket closed
  • VS Code Servers detects disconnection and starts a 3 hours grace period for reconnections
• 2 AM VS Code Server closes Extension Host and Management connections then shutdown since there was not reconnection intent (local compute is still sleeping)
• 8 AM
  • Local compute wakes
  • Disconnection is detected and reconnection loop is started
  • Fails to reconnect since VS Code server is not running on remote.

To solve VS Code Server not running, we could install and start the server on each reconnection, but it won't resolve the reconnection because of the following:

• New VS Code Server will be installed and started
• Client will reconnect using the reconnection token registered in the previously killed VS Code Server instance
• Connection will fail with "Unknown reconnection token (never seen)"

Follow up with VS Code:

• If we start VS Code Server without --auto-shutdown, would that help in this case? What would be the cost/performance implications of having the server running indefinitely?

  • Response: No, to have a seamless reconnection the UI needs to find exactly the same remote extension host. The server shutdown is different to the extension host shutdown, we can keep the server running indefinitely (which is not costly) but the extension host will burn out a lot of RAM (it might have language servers running and other stuff) thus it is killed after being disconnected for 3 hours.

• Once this condition is reached, will customer lose it's dirty files, editor state, etc?

  • Response: No, there should never be data loss. We need to make sure customer has "Hot Exit = onExit" setting.

Solution

• [ ] Recommend user to change setting Hot Exit to onExit in remote connection to avoid data loses
• [ ] Implement our own dialog when remote connection needs window reload to something like "Your remote connection was idle for long time, we've saved your data. [Continue]"