jiahao-shen commented 2 years ago

Environment data

VS Code version: 1.63.2
Jupyter Extension version: v2021.11.1001550889
Python Extension version: v2021.12.1559732655
OS (Windows | Mac | Linux distro) and version: MacOS
Python and/or Anaconda version: 3.7.3
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Jupyter server running: Remote

Describe

When i use remote kernel in vscode-jupyter, it cannot use the relative path, the Notebook File Root is not work for ${fileDirname}

amunger commented 2 years ago

Can you provide more details about what behavior you are experiencing vs what you expect? If you run a cell with the following, do you get the location of the notebook that you're currently running?

import os
print(os.getcwd())

jiahao-shen commented 2 years ago

I run the remote kernel by using command line jupyter notebook and set the remote kernel in vscode according to the uri. I run the same code in browser and vscode, and this is the different results.

amunger commented 2 years ago

OK, I can repro. It looks like ${filenameDir} is returning the location of the remote kernel, rather than the location of the notebook.

greazer commented 2 years ago

There's a minor thing we can do here for remote running on localhost. Since we know the directory is probably accessible on the localhost we could special case the situation.

If the remote is truly remote, it's a more complex problem to deal with.

greazer commented 2 years ago

@jiahao-shen, can you provide us some information as to why you're using remote features to connect to your localhost? Thanks!

StefanBrand commented 2 years ago

Our use case is a JupyterHub instance that spawns a personal JupyterLab in a Kubernetes cluster. Some files are only accessible in mounted directories inside the Kubernetes pod. Therefore I am:

Attaching to the pod via the Kubernetes extension
Connecting to the remote Jupyter Server via the Jupyter extension

Workaround

Log in to JupyterHub and start the kernel from there by opening the respective notebook. It will be started in the directory of the notebook.
Choose the new kernel from the dropdown in VSCode (labelled "1 existing connections")

TTTPOB commented 2 years ago

7976

maybe these two issues are talking about the same thing?

rchiodo commented 2 years ago

@TTTPOB I agree. Remote servers cannot use jupyter.notebookFileRoot because that value doesn't make any sense. For localhost, we can probably make an exception, but for a true 'remote' server this will never work.

TTTPOB commented 2 years ago

@TTTPOB I agree. Remote servers cannot use jupyter.notebookFileRoot because that value doesn't make any sense. For localhost, we can probably make an exception, but for a true 'remote' server this will never work.

yes I think we can definitely make an exception for a localhost jupyter server

I might suggest that simply offer a regex or something to let user determine whether to treat the jupyter server as a local server, here is a common situation:

A low performance server A, with like 100G RAM and 16 cores; A high performance workstation B, with like 1T RAM and 144 cores; They have mounted the same filesystem

users sshed into the server A, and launched vscode there, but want to let the server running on workstation B. (and @StefanBrand seems have a more complicated version of this)

so I think instead of hardcode the localhost server into the extension, a configurable rule may be more suitable. (or I might have to use socat to make a fake local server)

And also the extension let pylance treat the import statement as relative to the file (#8285 ) , will this be a source of inconsistancy?

I just encountered this inconsistency that I saw the relative import worked fine so I thought the jupyter kernel is running in the ${fileDirname}, but it turns out it is running on the dir where I starts the jupyter server.

So, if the "treat as local rule" feature implemented as last, I think pylance should obey the same rule as well

rchiodo commented 2 years ago

I'm wondering how we might do this. Remote is rather hard to get the 'relative' ness of working directories correct because the same file system isn't on both machines. But maybe if the user specified a 'local' as 'remote' rule we could assume you started your remote server at some location? Maybe you give it to us? Then from there we make an assumption that's the same relative path as the workspace?

rchiodo commented 2 years ago

Pylance is a separate issue. It has to live where your source files are. It needs access to the file system in order to parse them. And in order for it to 'live' somewhere, there needs to be a VS code extension host running there (either through remote ssh or desktop)

TTTPOB commented 2 years ago

But maybe if the user specified a 'local' as 'remote' rule we could assume you started your remote server at some location? Maybe you give it to us? Then from there we make an assumption that's the same relative path as the workspace?

Y, I think so. for the example above, workstation B has an ip address of 192.168.7.2, so I want all jupyter servers on workstation B treated like "local" servers, because server A and workstation B have the same filesystem (let user decide this part) And to make workstation B to be treated as a "local" server, we enable a option like Treat as Local Regex, and fill it in with ^http://192\.168\.7\.1.* I think the extension won't have to decide whether the remote server and the current machine has the same filesystem (it's rather difficult! ), let user define it.

StefanBrand commented 2 years ago

I do not completely follow the examples with the IP addresses. For my use-case, I'm attached to a Kubernetes pod. VSCode is aware of the file system because it let's me choose a workspace folder E.g. when I attach to the pod, I choose /home/jovyan/my-project as the workspace folder. Inside the workspace there would be two modules, one importing functions from the other.

axsk commented 1 year ago

Persistent notebook sessions are another usecase for remote-connetected local servers. Here I would like to start the kernel in my VS Code dir, this is not working atm.

DonJayamanne commented 1 year ago

Suggestion: Make it clear the remote file paths and the like do not work for remote kernels.

groldo commented 1 year ago

I have a slightly different setup but the problem seems to be the same:

I use a devcontainer setup, have the jupyter extension installed there and connect to a local kernel. Location is set to filenameDir.

When I create a new jupyter notebook via code (not saved to disk up to now) and run this code:

import os
print(os.getcwd())

I do get /workspace like expected

output from vs code jupyter extension:

info 09:40:27.182: Starting Jupyter Session startUsingPythonInterpreter, .jvsc74a57bd0d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe./opt/conda/python./opt/conda/python.-m#ipykernel_launcher (Python Path: /opt/conda, EnvType: Conda, EnvName: 'base', Version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:26:04) [GCC 10.4.0]) for 'Untitled-1.ipynb' (disableUI=false)
info 09:40:27.187: Computing working directory for resource 'Untitled-1.ipynb'
info 09:40:27.338: Process Execution: > /opt/conda/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> /opt/conda/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
info 09:40:27.519: Process Execution: > /opt/conda/bin/python -m pip list
> /opt/conda/bin/python -m pip list
info 09:40:27.805: Process Execution: > /opt/conda/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"df65e3cf-41e8-4a0a-9034-fae423ab4419" --shell=9007 --transport="tcp" --iopub=9009 --f=/home/jovyan/.local/share/jupyter/runtime/kernel-v2-359RVd5OXzL7SgM.json
> /opt/conda/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"df65e3cf-41e8-4a0a-9034-fae423ab4419" --shell=9007 --transport="tcp" --iopub=9009 --f=/home/jovyan/.local/share/jupyter/runtime/kernel-v2-359RVd5OXzL7SgM.json
info 09:40:27.805: Process Execution: cwd: /workspace
cwd: /workspace
info 09:40:28.284: ipykernel version & path 6.19.2, /opt/conda/lib/python3.10/site-packages/ipykernel/__init__.py for /opt/conda/bin/python
warn 09:40:30.912: StdErr from Kernel Process /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2544: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(

warn 09:40:30.921: StdErr from Kernel Process /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2495: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'df65e3cf-41e8-4a0a-9034-fae423ab4419' instead of 'b"df65e3cf-41e8-4a0a-9034-fae423ab4419"'.
  warn(

info 09:40:30.942: Kernel Output: NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049

info 09:40:30.944: Kernel Output: To connect another client to this kernel, use: --existing kernel-v2-359RVd5OXzL7SgM.json

info 09:40:31.201: Started Kernel base (Python 3.10.8) (pid: 2276)
error 09:40:31.239: stat() failed for "untitled:Untitled-1.ipynb?jupyter-notebook" [Error (FileSystemError): Der Dateisystemanbieter mit relativem Dateipfad "untitled:Untitled-1.ipynb?jupyter-notebook" konnte nicht aufgelöst werden.
    at Function.c (/vscode/vscode-server/bin/linux-x64/1ad8d514439d5077d2b0b7ee64d2ce82a9308e5a/out/vs/workbench/api/node/extensionHostProcess.js:96:26282)
    at Object.stat (/vscode/vscode-server/bin/linux-x64/1ad8d514439d5077d2b0b7ee64d2ce82a9308e5a/out/vs/workbench/api/node/extensionHostProcess.js:96:24413)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async b.exists (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:7:20750)
    at async b.getWorkingDirectory (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:2:2083864)
    at async u.getCode (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:2:2078752)
    at async Promise.all (index 1)
    at async t.Kernel.gatherInternalStartupCode (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:2:2038671)
    at async t.Kernel.initializeAfterStart (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:2:2037418)
    at async t.Kernel.createJupyterSession (/home/jovyan/.vscode-server/extensions/ms-toolsai.jupyter-2022.11.1003412109/out/extension.node.js:2:2034663)] {
  code: 'Unknown'
}
info 09:40:36.017: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8821ms
info 09:40:36.210: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8688ms
info 09:40:36.316: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8924ms
info 09:40:36.400: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8878ms

Now I save the notebook to disk. VS Code automatically opens the new file:

info 09:42:53.002: Starting Jupyter Session startUsingPythonInterpreter, .jvsc74a57bd0d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe./opt/conda/python./opt/conda/python.-m#ipykernel_launcher (Python Path: /opt/conda, EnvType: Conda, EnvName: 'base', Version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:26:04) [GCC 10.4.0]) for '/workspace/test56.ipynb' (disableUI=false)
info 09:42:53.006: Computing working directory for resource '/workspace/test56.ipynb'
info 09:42:53.089: Process Execution: > /opt/conda/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> /opt/conda/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
info 09:42:53.370: Process Execution: > /opt/conda/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"f17db154-82c2-458a-ae60-576e33fa8248" --shell=9007 --transport="tcp" --iopub=9009 --f=/home/jovyan/.local/share/jupyter/runtime/kernel-v2-359xMhjnim1f0j0.json
> /opt/conda/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"f17db154-82c2-458a-ae60-576e33fa8248" --shell=9007 --transport="tcp" --iopub=9009 --f=/home/jovyan/.local/share/jupyter/runtime/kernel-v2-359xMhjnim1f0j0.json
info 09:42:53.370: Process Execution: cwd: /workspace
cwd: /workspace
info 09:42:54.198: ipykernel version & path 6.19.2, /opt/conda/lib/python3.10/site-packages/ipykernel/__init__.py for /opt/conda/bin/python
warn 09:42:55.419: StdErr from Kernel Process /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2544: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(

warn 09:42:55.420: StdErr from Kernel Process /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2495: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'f17db154-82c2-458a-ae60-576e33fa8248' instead of 'b"f17db154-82c2-458a-ae60-576e33fa8248"'.
  warn(

info 09:42:55.475: Kernel Output: To connect another client to this kernel, use: --existing kernel-v2-359xMhjnim1f0j0.json

info 09:42:55.696: Started Kernel base (Python 3.10.8) (pid: 2980)
info 09:42:55.713: UpdateWorkingDirectoryAndPath in Kernel
info 09:42:58.844: Cancel all remaining cells true || Error || undefined
info 09:43:01.424: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8411ms
info 09:43:01.533: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8408ms
info 09:43:01.555: Got env vars with python /opt/conda/bin/python, with env var count 51 and custom env var count 48 in 8439ms

Run the code again and now it prints / for root directory. which is not expected.

pfarndt commented 1 year ago

I now managed to use VS Code to my (remote, tmux'ed) running jupyter-lab - I remotely run vs-code and connect to localhost:port. This keeps my notebook kernels alive even when I disconnect. All very nice - only this issues here, i.e. that notebook kernels start in the workspace dir but not in the subfolder, where the ipynb file is, holds me back from using vs-code for jupyter notebooks. Any help is appreciated.

DonJayamanne commented 1 year ago

@pfarndt You could try VS Code Server with tunnels See ehre https://code.visualstudio.com/docs/datascience/notebooks-web#_remote-tunneling-with-vs-code-server Basically this will let you run VS Code on the remote machine and have all of the same features you have as when running locally

pfarndt commented 1 year ago

@pfarndt You could try VS Code Server with tunnels See ehre https://code.visualstudio.com/docs/datascience/notebooks-web#_remote-tunneling-with-vs-code-server Basically this will let you run VS Code on the remote machine and have all of the same features you have as when running locally

I got the code tunnel to work - but here the same happens: as soon as I disconnect from the vs code tunnel, the kernel terminates. I understand that I would need a jupyter server running and then connect to a (remote) kernel. Problem here again: kernels start with with wrong cwd.

DonJayamanne commented 1 year ago

I disconnect from the vs code tunnel, the kernel terminates

Looks like you are after https://github.com/microsoft/vscode-jupyter/issues/3998 Please upvote that issue

thoth291 commented 1 year ago

For those who are interested, I discovered a very much undocumented way to make it work. My only hope is that developers, who will see it, won't be pissed off by me using this trick. And if they are, I also hope that they will excuse me and not change their internal api much for this to survive at least few seasons.

{
"jupyter.runStartupCommands": [
        "import os",
        "__t=os.path.dirname(__vsc_ipynb_file__)",
        "%cd {__t}",
        "del __t"
    ],
}

Maybe, you can like it - and they will accept it as DeFacto solution.

pfarndt commented 1 year ago

Thanks @thoth291 - this works great for kernels, which run asa child process of vs-code, and therefore die when the window is closed or the connection is otherwise lost.

However, one is also able to run a kernel as a child of an existing jupyter server as described here. Advantage: The jupyter server (if run in a multiplexer) and the kernel will not die once the window is closed.

In this case the above 4 lines of code are not executed. Is there a way to run this code also for kernels started as a child of a jupyter server?

DonJayamanne commented 1 year ago

y only hope is that developers, who will see it, won't be pissed off by me using this trick.

@thoth291 thanks for the great research and sharing this with the community, no we will not be annoyed by this work around. The variable is officially supported, though not documented as its meant for some internal tooling.

and they will accept it as DeFacto solution.

Unfortunately that will not work everwhere. I'm assuming you are connecting to localhost, if you were to connect to a remote kernel (something that is not running on your machine), then this would not work.

thoth291 commented 1 year ago

@pfarndt , @DonJayamanne - thank you for encouragement and support!

So that you would know - I'm connected to remote kernel and this does work. We are hosting our jupyter server setup using jupyter hub. Then I generate token for my default server at jupyter hub. And after selecting existing server as described in the settings shared above - it works perfectly well.

I also have to admit that I use remote dev extension through ssh - and I have setup my remote shell initialization in a way to see all available conda kernels.

Unfortunately I cannot share with you any videos - but trust me - since it does work on remote - and that was the whole reason for me to research it in the first place.

Another thing is that I NEVER use local jupyter kernel - all my work is on remote machines.

pfarndt commented 1 year ago

I confirm that the above workaround works for python kernels. I assume that the code is injected into a starting kernel, which then has the cwd as the dir of the ipynb file. However, I am a julia person and the above code does not work.

Even if I "inject" valid julia code:

{
"jupyter.runStartupCommands": [
        "qq=78",
        "cd(\"subdir\")"
    ],
}

it is not executed. Neither is the variable qq defined, nor the cwd changed. Is there a way to make this trick also work for julia?

To make this more general it would probably be nice to have an environment variable __vsc_ipynb_file__ which is set for all sorts of kernels to access.

iitrabhi commented 1 year ago

With the latest update __vsc_ipynb_file__ no longer works. It was working well before.

thoth291 commented 1 year ago

The variable is officially supported, though not documented as its meant for some internal tooling.

I'm using this version: Version: 1.77.1 (user setup) Commit: b7886d7461186a5eac768481578c1d7ca80e2d21 Date: 2023-04-04T23:21:11.906Z Electron: 19.1.11 Chromium: 102.0.5005.196 Node.js: 16.14.2 V8: 10.2.154.26-electron.0 OS: Windows_NT x64 10.0.22621 Sandboxed: Yes

And it works just fine - my extensions also up to date.

starball5 commented 8 months ago

Potentially related on Stack Overflow: How to set the file path of a remote jupyter notebook in VSCode?

microsoft / vscode-jupyter

Notebook File Root setting does not work with Remote Kernels #8771

Environment data

Describe

Workaround

7976