search

jupyter / jupyter_core

Core Jupyter functionality
https://jupyter-core.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
195 stars 181 forks source link

Jupyter in pyenv can't find subcommand when python is called directly #339

Closed bluecoconut closed 1 year ago

bluecoconut commented 1 year ago

I just set up a fresh mac (M2), and am running into issues running jupyter in vscode when my environment is set up with pyenv (homebrew version), and after digging a lot, i think the root cause is PATH manipulation when subcommands get executed.

Specifically:

Running /path/to/python -m jupyter works fine

❯ ~/.pyenv/versions/3.11.1/envs/jupyter-base/bin/python -m jupyter
usage: jupyter.py [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [--debug]
                  [subcommand]
...
Available subcommands: bundlerextension console dejavu events execute fileid kernel kernelspec lab labextension
labhub migrate nbclassic nbconvert nbextension notebook qtconsole run server serverextension troubleshoot trust

however, adding the subcommand fails. 

❯ ~/.pyenv/versions/3.11.1/envs/jupyter-base/bin/python -m jupyter notebook
pyenv: jupyter-notebook: command not found

The `jupyter-notebook' command exists in these Python versions:
  3.10.9/envs/research
  3.11.1/envs/jupyter-base
  jupyter-base
  research

I've tried adding a JUPYTER_PATH env as well as JUPYTER_PREFER_ENV_PATH=True (and a combo of both), but none of the above worked.

Also, since the python -m jupyter can see the subcommands, I would have assumed that the same enumeration logic there would have worked for calling the subcommand, so this feels like a bug in the jupyter_core package. But, let me know if I should raise this concern elsewhere.

bluecoconut commented 1 year ago

Okay, so i've resolved this in a few different ways -- and i think it's a mixed jupyter_core assumption and pyenv problem.

The core issue is that I didn't have a pyenv global set, so even though pyenv was setting up shims in the path, these shims were unresolved (no global context), and pyenv shorted these but didn't pass through to the rest of PATH, since it tries to resolve its own global context. This now "works" but it actually means that i might accidentally point the the wrong jupyter command if it's not the same as global.

That said, I would have expected when running python -m jupyter subcommand the subcommand is prioritized to be executed from scripts directory.

This feels like the same "prioritization" JUPYTER_PREFER_ENV_PATH is about -- so I put a quick PR that fixes this issue for me using that variable. I'm not sure this is actually the intent of the variable though...

bluecoconut commented 1 year ago

Code changes that made this work for me

https://github.com/jupyter/jupyter_core/pull/340

I guess in general, I'm questioning the decision to put the scripts at the end of path rather than at the start of path. I would have expected this to always be an insert(0 rather than .append

bluecoconut commented 1 year ago

Closing this for now - in the end the trick to help anyone who is struggling with this would be to go to _path_with_self() in commands.py, and print out the new resolve path that is generated, and see how the which command resolves https://github.com/jupyter/jupyter_core/blob/484d41278bde1ea3c64ac996ee4c7d0d42a803b3/jupyter_core/command.py#L142