Unable to connect to Kernel of updated remote Jupyter Server without restarting VSCode

rjra2611 commented 1 year ago

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and\/or Cell Scripts (.py files with #%% markers)

What happened?

Hey,

I have implemented the interface IJupyterUriProvider to connect my remote Jupyter servers.

Whenever I update my remote Jupyter server than in that case I'm unable to connect to the new remote kernels The connection to the new remote Jupyter server seems to be successful as I don't get any errors (which I do when I tried with wrong server uri) and getServerUri method is successfully resolved.

The only way I'm able to connect to the kernel of the updated remote Jupyter server is by restarting (closing and reopening) the VSCode.

Attaching the verbose logs for both the cases

without restart.txt After restart.txt

The difference I notice in the logs is that the After restart.txt logs contains

18:35:56.349 [debug] Disposing session manager

which doesn't happen without restart once both have logged No controller, hence notebook communications cannot be initialized

VS Code Version

Visual Studio Code (1.78.1, undefined, desktop)

Jupyter Extension Version

Jupyter Extension Version: 2023.4.1011241018.

Jupyter logs

Attached the Logs in the Issues Description

Coding Language and Runtime Version

No response

Language Extension Version (if applicable)

No response

Anaconda Version (if applicable)

No response

Running Jupyter locally or remotely?

None

DonJayamanne commented 1 year ago

Thanks for using the API and filing this issue.

If the Uri changes, are you triggering the onDidChangeHandles event in the IJupyterUriProvider? If not, then I would suggest doing that

rjra2611 commented 1 year ago

Hey, thanks for the quick response. As suggested I tried implementing the onDidChangeHandles event as follows

private _onDidChangeHandles = new EventEmitter<void>();

get onDidChangeHandles(): Event<void> {
    return this._onDidChangeHandles.event;
}

// Call this method whenever the list of server handles changes
public fireDidChangeHandles(): void {
    this._onDidChangeHandles.fire();
}

I fire the fireDidChangeHandles method whenever there I update the remote Jupyter server uri. But still can't connect to the new kernel

implement onChangeHandles.txt

Comparing the logs between successful connection v/s unsuccessful kernel connection

Unsuccessful connection

17:45:52.359 [debug] Disposing session manager
17:45:52.359 [debug] SessionManager - dispose contents manager
17:45:52.359 [debug] ShutdownSessionAndConnection - dispose session manager
17:45:52.359 [debug] Finished disposing jupyter session manager

Successful connection

17:51:04.190 [debug] Disposing session manager
17:51:04.190 [debug] SessionManager - dispose contents manager
17:51:04.190 [debug] ShutdownSessionAndConnection - dispose session manager
17:51:04.190 [debug] Finished disposing jupyter session manager
17:51:04.790 [debug] UniversalRemoteKernelFinder: get from cache
17:51:04.792 [debug] Checking for server usability.
17:51:04.792 [debug] Starting notebook server.
17:51:05.460 [debug] Connecting to process server
17:51:05.460 [info] Connecting remote server kernel http://localhost:9486/
17:51:05.463 [debug] Connection complete server
17:51:05.493 [debug] Disposing session manager
17:51:05.493 [debug] SessionManager - dispose contents manager
17:51:05.493 [debug] ShutdownSessionAndConnection - dispose session manager
17:51:05.494 [debug] Finished disposing jupyter session manager
17:51:05.494 [debug] UniversalRemoteKernelFinder: Writing 5 remote kernel connection metadata to cache
17:51:05.510 [info] End refreshing Kernel Picker (1683894039927)
17:51:05.831 [debug] _jupyterPythonPathFunction: Giving Pylance "d:\python-3.10\python.exe" as python path for "file:///d%3A/quantconnect/vscode-lean-init/Emotional%20Black%20Alpaca/research.ipynb"
17:51:05.834 [debug] KernelProvider switched kernel to id = a11dc20d3050651d8a82820f518d6c1b2fd29a40ebba4aea1bff78cc5ab9681f..python3.python.\.-m#ipykernel_launcher
17:51:05.834 [debug] Initialize notebook communications for editor d:\quantconnect\vscode-lean-init\Emotional Black Alpaca\research.ipynb
17:51:05.834 [debug] Resolving notebook UI Comms (resolve) for d:\quantconnect\vscode-lean-init\Emotional Black Alpaca\research.ipynb
17:51:05.834 [debug] initialize CommonMessageCoordinator
17:51:05.835 [debug] Registering commtarget jupyter.widget
17:51:05.835 [debug] IPyWidgetMessageDispatcher.initialize
17:51:05.835 [debug] Attempting to determine version of IPyWidgets
17:51:05.836 [debug] Controller selection change completed
17:51:05.836 [debug] Waiting for IPyWidgets version
17:51:05.836 [debug] Waiting for IPyWidgets version promise
17:51:20.524 [debug] Updating cache with Remote kernels

rjra2611 commented 1 year ago

Hey,

Can I ask if there are any updates on the issue? Happy to provide more debugging information if required.

DonJayamanne commented 1 year ago

Are you providing a new handle, ie is the handle different after you update the server or is it the same.

Also please could you test this against the latest pre release version in vscode insiders. I fixed a few issues in this area recently

rjra2611 commented 1 year ago

Are you providing a new handle, ie is the handle different after you update the server or is it the same.

Nope, the handle seems to be the same i.e what was provided initially when registering an instance of IJupyterUriProvider.

The new server uri (IJupyterServerUri) is provided when getServerUri is called. I believe IJupyterServerUri doesn't have an attribute to provide handle therefore, I believe the handle always remains the same.

Also please could you test this against the latest pre release version in vscode insiders. I fixed a few issues in this area recently

Got it, will try to test with https://code.visualstudio.com/insiders/#

DonJayamanne commented 1 year ago

If the server Uri changes, then please create a new handle and trigger a change instead of returning the new server Uri from the getServerUri method. Also ensure the old handle is no longer returned in the list of handles.

Try that and let me know how it goes

rjra2611 commented 1 year ago

Hey, Thanks

Triggering the handle change and sending the updated handle list does seem to trigger getServerUri but when trying to run the notebook with the kernel of the updated server then the notebook throws:

02:07:10.556 [debug] VSCodeNotebookController::handleExecution, Class name = Dy (started execution)
02:07:10.556 [debug] Execute Cell 1 d:\quantconnect\vscode-lean-init\Creative Yellow Green Hornet\research.ipynb
02:07:10.557 [debug] start the kernel, options.disableUI=false for d:\quantconnect\vscode-lean-init\Creative Yellow Green Hornet\research.ipynb
02:07:10.557 [debug] Execute Cell 2 d:\quantconnect\vscode-lean-init\Creative Yellow Green Hornet\research.ipynb
02:07:10.557 [debug] start the kernel, options.disableUI=false for d:\quantconnect\vscode-lean-init\Creative Yellow Green Hornet\research.ipynb
02:07:10.557 [debug] End cell 1 execution @ undefined, started @ undefined, elapsed time = 0s
02:07:10.557 [error] Error in execution Error: notebook controller is DISPOSED
    at Object.createNotebookCellExecution (c:\<username>s\<username>\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:100:97574)
    at Function.create (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194831)
    at Function.getOrCreate (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194711)
    at Dy.createCellExecutionIfNecessary (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200053)
    at Dy.executeCell (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200510)
    at async Promise.all (index 0)
    at Dy.handleExecution (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:197326)
02:07:10.558 [error] Error in execution (get message for cell) Error: notebook controller is DISPOSED
    at Object.createNotebookCellExecution (c:\<username>s\<username>\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:100:97574)
    at Function.create (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194831)
    at Function.getOrCreate (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194711)
    at Dy.createCellExecutionIfNecessary (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200053)
    at Dy.executeCell (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200510)
    at async Promise.all (index 0)
    at Dy.handleExecution (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:197326)
02:07:10.558 [error] Error in execution Error: notebook controller is DISPOSED
    at Object.createNotebookCellExecution (c:\<username>s\<username>\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:100:97574)
    at Function.create (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194831)
    at Function.getOrCreate (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194711)
    at Dy.createCellExecutionIfNecessary (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200053)
    at Dy.executeCell (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200510)
    at async Promise.all (index 1)
    at Dy.handleExecution (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:197326)
02:07:10.558 [error] Error in execution (get message for cell) Error: notebook controller is DISPOSED
    at Object.createNotebookCellExecution (c:\<username>s\<username>\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:100:97574)
    at Function.create (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194831)
    at Function.getOrCreate (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194711)
    at Dy.createCellExecutionIfNecessary (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200053)
    at Dy.executeCell (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200510)
    at async Promise.all (index 1)
    at Dy.handleExecution (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:197326)
02:07:10.562 [error] VSCodeNotebookController::handleExecution, Class name = Dy, completed in 6ms, has a falsy return value Error: notebook controller is DISPOSED
    at Object.createNotebookCellExecution (c:\<username>s\<username>\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\api\node\extensionHostProcess.js:100:97574)
    at Function.create (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194831)
    at Function.getOrCreate (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:194711)
    at gm (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:17:203807)
    at Dy.executeCell (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:200881)
    at async Promise.all (index 0)
    at Dy.handleExecution (c:\<username>s\<username>\.vscode\extensions\ms-toolsai.jupyter-2023.4.1011241018-win32-x64\out\extension.node.js:24:197326)

DonJayamanne commented 1 year ago

Ok I think understand what's happening Questions

Why is the server changing, ie why do you need to start a new server

The way the UI works is, if the server/handle changes, then the previously selected notebook kernel is deleted/disposed and the user will need to select another kernel and run a cell

What is different between the new and the old server. Are the tokens/auth details different

rjra2611 commented 1 year ago

Hey, thanks for the quick response!

Why is the server changing, ie why do you need to start a new server

We have servers running on CPU/GPU and users toggle between them.

What is different between the new and the old server. Are the tokens/auth details different

Yup, both server uri and auth tokens are different

The way the UI works is, if the server/handle changes, then the previously selected notebook kernel is deleted/disposed and the user will need to select another kernel and run a cell

The way I'm tackling this is on server uri change:

Close all the open notebooks to dispose old kernel

for (const editor of editors) {
    await vscode.window.showNotebookDocument(editor.notebook);
    await vscode.commands.executeCommand('workbench.action.closeActiveEditor');
}

use openNotebookDocument API to open and connect with the new kernel as you suggested earlier

The new kernel does reflect on the notebook but when I run the notebook it throws with the error shared in the previous comment

rjra2611 commented 1 year ago

Even if I don't close the notebooks and use the openNotebookDocument API to switch the new kernels. It seems to successfully connect to the new kernel ( I can see it going from select kernel to my new kernel) and also the below logs seems to suggest that it connected to the new kernel but I get the error as mentioned before on running the notebook cell

03:30:22.453 [info] Connecting remote server kernel <server-uri>
03:30:22.456 [debug] Connection complete server
03:30:24.105 [debug] Disposing session manager
03:30:24.105 [debug] SessionManager - dispose contents manager
03:30:24.105 [debug] ShutdownSessionAndConnection - dispose session manager
03:30:24.105 [debug] Finished disposing jupyter session manager
03:30:24.105 [debug] UniversalRemoteKernelFinder: Writing 10 remote kernel connection metadata to cache
03:30:24.117 [info] End refreshing Kernel Picker (1684360601494)

DonJayamanne commented 1 year ago

Looks like you are not using the latest pre-release version of the Jupyter extension, please can you test this again the latest pre-release version of the Jupyter extension and vs code insiders

Please install latest VS Code Insiders
Open the extensions page, search for Jupyter and click the button Switch to Pre-Release Version
Similarly please install the latest Pre-Release version of the Python extension

both server uri

Sorry to ask this again, but is host name and port number different or is part of the query string different in the Uri.
Finally, after you have implemented the new changes and installed vscode insiders and the latest pre-release, please can you try this again and share the logs. Thanks, still trying to figure out where this is going wrong. thanks for your patience

rjra2611 commented 1 year ago

Hey,

Did a test with VSCode insiders, Jupyter Pre Release, and Python Pre Release but got the same bug.

VSCode Insider Debug logs.txt

is host name and port number different or is part of the query string different in the Uri.

No, it's part of the uri itself.

DonJayamanne commented 1 year ago

I beleive I've identified the issues here, unfortunately the fixes will take a while to get into the product as there is quite a lot that needs to be fixed in this space. I have a PR with the changes, but trying to find a safe way to ship these changes in the next releaes. Will keep you posted

NightMachinery commented 1 year ago

https://github.com/microsoft/vscode-jupyter/issues/13849

DonJayamanne commented 1 year ago

@rjra2611 plese can you test the latest relese version, I believe this should be fixed now.

DonJayamanne commented 1 year ago

Closing for now, if this issue persists, please feel free to create a new issue. You might also be interested in this issue https://github.com/microsoft/vscode-jupyter/issues/13894

We're looking at making the API stable and simplifying it.

roblourens commented 1 year ago

Looks like there is nothing to verify here @DonJayamanne ?

microsoft / vscode-jupyter