Closed quantum-byte closed 4 months ago
DonJayamanne
commented
4 months ago Thank you very much for the detailed analysis, and providing the great sample for being able to easily repro this. I will take a look at this soon, and will try to get this fixed for the next release (unfortunately we will be shipping a new version next week & it is too late to get this fixed for that).
DonJayamanne
commented
4 months ago @quantum-byte Please can you install this version of Jupyter Hub extension, & let me know if this works for you. Tested at my end and works (again thanks for the great steps)
quantum-byte
commented
4 months ago Hi @DonJayamanne , thanks for looking into it so quickly!
Please can you install this version of Jupyter Hub extension, & let me know if this works for you. Tested at my end and works (again thanks for the great steps)
I tested it and i can execute my Test Notebook successfully as far as i can see. But i still see a few Errors in the vscode-jupyter extension output regarding some extension.js not being able to be downloaded (see below). Not sure if this is related or something else entirely.
As an example for one of the urls if i use the adjusted http://localhost:8000/user/user%20%28test%29/nbextensions/jupyterlab-plotly/extension.js i get the following successful response:
// Entry point for the notebook bundle containing custom model definitions.
//
define(function() {
"use strict";
window['requirejs'].config({
map: {
'*': {
'jupyterlab-plotly': 'nbextensions/jupyterlab-plotly/index',
},
}
});
// Export the required load_ipython_extension function
return {
load_ipython_extension : function() {}
};
});I am also quite surprised, that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).
Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)
``` Visual Studio Code (1.87.2, undefined, desktop) Jupyter Extension Version: 2024.2.0. Python Extension Version: 2024.2.1. Pylance Extension Version: 2024.3.2. Platform: linux (x64). Multiple Workspace folders opened <**********> 10:56:41.124 [info] Start refreshing Kernel Picker (1711619801124) 10:56:41.138 [info] Using Pylance 10:56:42.828 [info] Start refreshing Interpreter Kernel Picker 10:56:45.913 [info] Process Execution: ~/.mambaforge/envs/testenv/bin/python -m pip list 10:56:46.041 [info] No interpreter for Pylance for Notebook URI "<******|.ipynb" 10:56:51.631 [info] End refreshing Kernel Picker (1711619801124) 10:56:55.443 [info] Start refreshing Kernel Picker (1711619815443) 10:56:55.682 [info] End refreshing Kernel Picker (1711619815443) 10:56:59.198 [info] Handle Execution of Cells 5,8,10,14,15,20,22,30 for <******|.ipynb 10:56:59.200 [info] Starting Kernel startUsingRemoteKernelSpec, 7e801c78ddcea37e824adca34211c78ef7f7527d1106c3d51cad9b3a0be0e7a5..testenv./opt/conda/envs/testenv/python./.-m#ipykernel_launcher for '<******|.ipynb' (disableUI=false) 10:56:59.256 [info] http://localhost:8000/user/user (test)/: Kernel gestartet: 4a9cd58b-cfbd-48e5-81d6-627bc3fbca75 10:56:59.782 [info] Started session for kernel startUsingRemoteKernelSpec:7e801c78ddcea37e824adca34211c78ef7f7527d1106c3d51cad9b3a0be0e7a5..testenv./opt/conda/envs/testenv/python./.-m#ipykernel_launcher 10:56:59.900 [info] Registering Kernel Completion Provider from kernel Python 3 (testenv) for language python 10:56:59.900 [info] Start refreshing Kernel Picker (1711619819900) 10:57:00.035 [info] Kernel acknowledged execution of cell 5 @ 1711619820034 10:57:00.248 [info] End refreshing Kernel Picker (1711619819900) 10:57:01.504 [info] End cell 5 execution after 1.47s, completed @ 1711619821504, started @ 1711619820034 10:57:01.535 [info] Kernel acknowledged execution of cell 8 @ 1711619821535 10:57:01.538 [error] Error downloading from http://localhost:8000/user/user (test)/nbextensions/jupyter-js-widgets/extension.js: Not Found 10:57:01.538 [warn] Failed to fetch script sources [TypeError: Cannot read properties of undefined (reading 'replace') at Hc.ourFetch [as fetchImplementation] (~/.vscode/extensions/ms-toolsai.jupyter-hub-2024.4.100/dist/extension.node.js:68102:32) at Hc.downloadFile (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:265:13637) at UT.getWidgetScriptSourceUsingFetch (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:341:86140) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)] 10:57:01.541 [error] Error downloading from http://localhost:8000/user/user (test)/nbextensions/jupyterlab-plotly/extension.js: Not Found 10:57:01.777 [info] End cell 8 execution after 0.242s, completed @ 1711619821777, started @ 1711619821535 10:57:01.788 [info] Kernel acknowledged execution of cell 10 @ 1711619821788 10:57:01.814 [info] End cell 10 execution after 0.026s, completed @ 1711619821814, started @ 1711619821788 10:57:01.821 [info] Kernel acknowledged execution of cell 14 @ 1711619821821 10:57:01.879 [info] End cell 14 execution after 0.058s, completed @ 1711619821879, started @ 1711619821821 10:57:01.886 [info] Kernel acknowledged execution of cell 15 @ 1711619821886 10:57:09.512 [info] End cell 15 execution after 7.626s, completed @ 1711619829512, started @ 1711619821886 10:57:09.518 [info] Kernel acknowledged execution of cell 20 @ 1711619829518 10:57:09.748 [info] End cell 20 execution after 0.23s, completed @ 1711619829748, started @ 1711619829518 10:57:09.753 [info] Kernel acknowledged execution of cell 22 @ 1711619829753 10:57:12.043 [info] End cell 22 execution after 2.29s, completed @ 1711619832043, started @ 1711619829753 10:57:12.049 [info] Kernel acknowledged execution of cell 30 @ 1711619832049 10:57:12.558 [info] End cell 30 execution after 0.509s, completed @ 1711619832558, started @ 1711619832049 ```
Regarding
(unfortunately we will be shipping a new version next week & it is too late to get this fixed for that)
That is fine. For now we can tell the small amount of affected users to downgrade to the older extension version.
DonJayamanne
commented
4 months ago that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).
Your assessment was accurate, however I didn't want to modify the Jupyter extension to get around the issue with the Jupyter Hub problem.
Here's the solution that keeps this fix in the Jupyter Hub extension https://github.com/microsoft/vscode-jupyter-hub/pull/52 This way Url encoding thats specific to Hub stays there, and other Urls are treated differently.
not being able to be downloaded (see below). Not sure if this is related or something else entirely.
You can ignore this logged error, its a noop, it should work, please test and let me know if plotly widgets work/or not. From what I can tell it should work and the error thats logged is a false positive (will modify to not log this)
quantum-byte
commented
4 months ago that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).
Your assessment was accurate, however I didn't want to modify the Jupyter extension to get around the issue with the Jupyter Hub problem.
Here's the solution that keeps this fix in the Jupyter Hub extension #52 This way Url encoding thats specific to Hub stays there, and other Urls are treated differently.
Right, that totally makes sense! Fix it where the responsibility lies.
not being able to be downloaded (see below). Not sure if this is related or something else entirely.
You can ignore this logged error, its a noop, it should work, please test and let me know if plotly widgets work/or not. From what I can tell it should work and the error thats logged is a false positive (will modify to not log this)
Its not quite a noop as far as i can tell. In the end it works in both cases since the fallback is to download the extension code from the cdn.
But it still is a inconsistent behaviour since for "normal" usernames without special characters the extension code is downloaded from the Jupyterhub instance itself instead of the cdn. I opened a separate Ticket for it: https://github.com/microsoft/vscode-jupyter-hub/issues/55
With newer
vscode-jupyterextension versions thanv2023.9.1102792234, it is no longer possible to access / use Jupyterhub kernels for Jupyterhub users with special characters in their username likeuser (test).Its the same issue somebody raised in issue 15055 as far as i can see but this one was closed.
This issue is also not in the separate
ms-toolsai.jupyter-hubextension. The authentication works fine but the subsequent kernel iteration throughvscode-jupyterbreaks.Environment data
v2023.9.1102792234e.g.v2024.2.0Expected behaviour
The specific Jupyterhub user can login via the web ui and their server instance is started. They then can use the Jupyterhub Server URL, username and generated token to setup a new Jupyterhub connection to access and use the kernels there in VSCode. Finally they can use one of the shown kernels to execute a example Notebook.
This also works if one uses
vscode-jupyterextension versionsv2023.9.1102792234.Actual behaviour
The specific Jupyterhub user can login via the web ui and their server instance is started. They then can use the Jupyterhub Server URL, username and generated token to setup a new Jupyterhub connection to access and use the kernels there in VSCode.
Get an error
Cant connect to Remoteserverafter the Jupyterhub connection setup is complete and i cant see any kernels from the remote server to use. You can find the complete Extension Output at the bottom.The relevant line from the output is
http://localhost:8000/user/user%20(test)/api/kernels?1711536811205. What works and what the Jupyterhub expects ishttp://localhost:8000/user/user%20%28test%29/api/kernels?1711536811205.I am unsure if the error is only due to our use of the Jupyterhub separate proxy deployment or if the Jupyterhub internal proxy also has this issue, but it is very likely.
The fact is that jupyterhub encodes every special character except
~and@for the username and expects the username to be encoded in request paths like that. This can partially be seen here and here. For some background behind this behaviour i found also this issue which talks about it.Basically what happens then is, that the Jupyterhub proxy knows to redirect a request for
http://localhost:8000/user/user%20%28test%29/but not forhttp://localhost:8000/user/user%20(test)/. That is why we see the 404 error in the log output.My best guess is you broke this with https://github.com/microsoft/vscode-jupyter/commit/4b47894e51122f1552803df87aef5043ce6a3a07 by no longer encoding some characters in the whole base url which also contains the username. But i think it was also not fully consistent with how Jupyterhub expects everything to be encoded before.
Also as far as i could see the
vscode-jupyterhubextension is not affected by this issue because they retrieve the user infoserverproperty from the Jupyterhub server and build the server user instance url by appending it to the base url as can be seen here (At least in the happy path where the Jupyter server returns theserverproperty for the user. The fallback implementation looks buggy). The Jupyterhub user infoserverproperty contains the correctly encoded username.Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]
Setup or use a local or remote Jupyterhub instance. Login in and start the server with a user called
user (test). This is quite easy to simulate with theDummyAuthenticatorauthenticator module which allows any kind of username to be tested.Setup a API Tokens for authentication on the Jupyterhub instance
Use the url, user name and token and create a new remote Jupyterhub connection
existing Jupyterhub-Serverenter URL for existing Serverhttp://localhost:8000user (test)JupyterHub (test)Cant connect to Remoteserver. See also the image (its in german though)Logs
Output for
Jupyterin theOutputpanel (View→Output, change the drop-down the upper-right of theOutputpanel toJupyter)``` 11:53:31.116 [info] Start refreshing Kernel Picker (1711536811116) 11:53:31.211 [error] Error fetching kernels from http://localhost:8000/user/user (test)/ (JupyterHub (test)): [r [Error]: Invalid response: 404 Not Found at r.create (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:1067) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.Rl [as listRunning] (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:4507) at async r.getRunningKernels (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12975) at async Promise.all (index 0) at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:19197) at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485] { response: p [Response] { size: 0, timeout: 0, [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null }, [Symbol(Response internals)]: { url: 'http://localhost:8000/user/user%20(test)/api/kernels?1711536811205', status: 404, statusText: 'Not Found', headers: [p [Headers]], counter: 0 } }, traceback: '' } 11:53:31.213 [error] UniversalRemoteKernelFinder: Failed to get kernels without cache [r [Error]: Invalid response: 404 Not Found at r.create (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:1067) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.Rl [as listRunning] (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:4507) at async r.getRunningKernels (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12975) at async Promise.all (index 0) at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:19197) at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485] { response: p [Response] { size: 0, timeout: 0, [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null }, [Symbol(Response internals)]: { url: 'http://localhost:8000/user/user%20(test)/api/kernels?1711536811205', status: 404, statusText: 'Not Found', headers: [p [Headers]], counter: 0 } }, traceback: '', vslsStack: [ CallSite {}, CallSite {}, CallSite {}, CallSite {}, CallSite {}, CallSite {}, CallSite {} ] } 11:53:31.214 [error] Jupyter Lab Helper:getKernelSpecs failure: [Error: Poll (@jupyterlab/services:KernelSpecManager#specs) is disposed. at v.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:10935) at ni.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:29299) at r.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12579) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 0) at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:20060) at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485] 11:53:31.237 [info] End refreshing Kernel Picker (1711536811116) ```