Feature request: jupyter command searches in Python's "scripts" directory for subcommands

[pfmoore]

My use case here is fairly unusual, so it may be that this is something you don't want to support. But what I am trying to do is as follows. I want to keep my Jupyter environment isolated from my system Python, so I install it into a virtual environment. If I do that, and run jupyter notebook, either with the venv activated, or using the full path to the jupyter.exe command, everything works as expected.

However, I would rather not need to activate the environment, or use the full path (I don't want to accidentally run the venv python, and typing the full path on the command line is a pain). I could use an alias for that, but what I usually do is copy the script wrapper (jupyter.exe) to my ~/.local/bin directory. This works for other applications, but jupyter fails to find its subcommands if I do this.

The problem seems to be in jupyter_core.command in function _path_with_self, which adds the directory containing the executable to PATH, for looking up subcommands. That fails if the jupyter.exe program gets moved.

A fix which seems to work for me is to add sysconfig.get_path('scripts') to the list of paths added to PATH. A proof of concept fix to _path_with_self is as follows:

def _path_with_self():
    """Put `jupyter`'s dir at the front of PATH

    Ensures that /path/to/jupyter subcommand
    will do /path/to/jupyter-subcommand
    even if /other/jupyter-subcommand is ahead of it on PATH
    """
    path_list = (os.environ.get('PATH') or os.defpath).split(os.pathsep)

    # Insert the "scripts" directory for this Python installation
    import sysconfig
    path_list.insert(0, sysconfig.get_path('scripts'))

    scripts = [sys.argv[0]]
    # Rest is unchanged

Is this something that could be considered for inclusion? I don't think it will cause problems anywhere else, and even though my use case is unusual, looking in the scripts location is a fairly natural and reasonable thing to do anyway.

I can create a PR for this if there is interest.

[EFanZh]

I am using Python 3.8 installed from Microsoft Store. I noticed that the directory returned by sysconfig.get_path('scripts') does not exist, but the directory returned by sysconfig.get_path('scripts', 'nt_user') exists, and with proper executables in it. Since sysconfig.get_path('scripts') doesn’t exist, I couldn’t launch Jupyter Notebook.

[pfmoore]

Hmm, it sounds like the store build has a rather strange setup. Does patching jupyter_core/commands.py as follows fix this?

diff --git a/jupyter_core/command.py b/jupyter_core/command.py          
index b9c60d2..8f8a63d 100644                                           
--- a/jupyter_core/command.py                                           
+++ b/jupyter_core/command.py                                           
@@ -164,7 +164,8 @@ def _path_with_self():                              
         # The Python environment does not specify a "scripts" location 
         pass                                                           
     else:                                                              
-        path_list.append(bindir)                                       
+        if bindir is not None:                                         
+            path_list.append(bindir)                                   

     scripts = [sys.argv[0]]                                            
     if os.path.islink(scripts[0]):                                     

[EFanZh]

Sorry I didn’t make myself clear. I mean on my PC, running sysconfig.get_path('scripts'), I get

C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.240.0_x64__qbz5n2kfra8p0\Scripts

And this directory does not exist.

Running sysconfig.get_path('scripts', 'nt_user'), I get:

C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts

This directory exists, and contains “jupyter-notebook.exe” in it.

To my understanding, those two directories are like system scripts path and user scripts path. The Python from Microsoft Store version doesn’t have system scripts path. Instead, using pip install (both with and without --user flag) command, the package executables are somehow installed into user scripts path. And current implementation doesn’t search user scripts path.

[pfmoore]

In that case, surely everything will still work fine, as the code will still search alongside jupyter.exe, which is where the subcommands will be in a normal install?

Since sysconfig.get_path('scripts') doesn’t exist, I couldn’t launch Jupyter Notebook.

As the directory doesn't exist, it should just get ignored (if I understand the code that uses the path list). Can you explain exactly how launching the notebook fails? What error (and traceback) do you get?

[EFanZh]

Traceback:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.240.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.240.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\jupyter.py", line 4, in <module>
    main()
  File "C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\jupyter_core\command.py", line 247, in main
    command = _jupyter_abspath(subcommand)
  File "C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\jupyter_core\command.py", line 133, in _jupyter_abspath
    raise Exception(
Exception: Jupyter command `jupyter-notebook` not found.

The code tries to find an executable named “jupyter-notebook.exe”, but it is in the user scripts path. The current implementation only searches system scripts path, and it can’t find the executable, so the exception is raised.

[pfmoore]

Is the jupyter command itself not in the user scripts path?

The current implementation only searches system scripts path

It doesn't (as far as I know) only search the system scripts path, rather it searches the system path in addition to the previously searched directories (the contents of PATH and the location of the jupyter script). So unless you moved your jupyter command, this specific feature should have had no effect on you.

Note: If you are saying that you'd like an additional change to search the user scripts path as well as the system scripts path, then I see no problem with that, although it should probably be a separate request and someone would need tow rite a PR. My concern here is that you seem to be saying that this change has broken a previously working scenario, which the PR I submitted should not have done.

[EFanZh]

No, your PR didn’t break anything. I installed Jupyter Notebook, and couldn’t start it, so I did a little digging. I guess the problem is because the correct scripts path not being searched, and this issue is about adding scripts path to search paths, so I commented my findings here, hoping my findings could help fixing the problem.

[pfmoore]

OK cool. I'm still not clear why the existing code doesn't find jupyter-notebook.exe, as that should be in the same directory as jupyter.exe. Maybe that's because store apps run in a protected context somehow, and the directory search is failing in a way that it doesn't with a "normal" directory? You may need to raise this as an issue on the Python bug tracker (https://bugs.python.org) as it looks like it may be related to the store build.

@zooba is the expert on the store build - maybe he can comment?

[EFanZh]

jupyter-notebook.exe is in the same directory as jupyter.exe, they are both in the user scripts directory.

C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts>dir
 Volume in drive C has no label.
 Volume Serial Number is BE8E-499E

 Directory of C:\Users\EFanZh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts

2019-11-02  17:39    <DIR>          .
2019-11-02  17:39    <DIR>          ..
2019-11-02  13:03               495 autopep8-script.py
2019-11-02  13:03            74,752 autopep8.exe
2019-11-02  13:03           103,349 epylint.exe
2019-11-02  17:20           103,361 iptest.exe
2019-11-02  17:20           103,361 iptest3.exe
2019-11-02  17:20           103,354 ipython.exe
2019-11-02  17:20           103,354 ipython3.exe
2019-11-02  13:03           103,339 isort.exe
2019-11-02  17:21           103,347 jlpm.exe
2019-11-02  17:20           103,343 jsonschema.exe
2019-11-02  17:21           103,363 jupyter-bundlerextension.exe
2019-11-02  17:20           103,353 jupyter-kernel.exe
2019-11-02  17:20           103,391 jupyter-kernelspec.exe
2019-11-02  17:21           103,346 jupyter-lab.exe
2019-11-02  17:21           103,353 jupyter-labextension.exe
2019-11-02  17:21           103,349 jupyter-labhub.exe
2019-11-02  17:17           103,349 jupyter-migrate.exe
2019-11-02  17:20           103,351 jupyter-nbconvert.exe
2019-11-02  17:21           103,350 jupyter-nbextension.exe
2019-11-02  17:21           103,349 jupyter-notebook.exe
2019-11-02  17:20           103,370 jupyter-run.exe
2019-11-02  17:21           103,354 jupyter-serverextension.exe
2019-11-02  17:17           103,354 jupyter-troubleshoot.exe
2019-11-02  17:20           103,382 jupyter-trust.exe
2019-11-02  17:17           103,349 jupyter.exe
2019-10-19  09:16           103,350 pip.exe
2019-10-19  09:16           103,350 pip3.8.exe
2019-10-19  09:16           103,350 pip3.exe
2019-11-02  13:03           103,342 pycodestyle.exe
2019-11-02  17:20           103,345 pygmentize.exe
2019-11-02  17:21           103,339 pyjson5.exe
2019-11-02  13:03           103,347 pylint.exe
2019-11-02  13:03           103,353 pyreverse.exe
2019-11-02  17:17            24,815 pywin32_postinstall.py
2019-11-02  17:17             3,335 pywin32_testall.py
2019-11-02  13:03           103,349 symilar.exe
2019-11-02  17:17    <DIR>          __pycache__
              36 File(s)      3,410,693 bytes
               3 Dir(s)  353,235,083,264 bytes free

[EFanZh]

From https://jupyter.org/install:

If installing using pip install --user, you must add the user-level bin directory to your PATH environment variable in order to launch jupyter lab.

I think this is also because of user scripts directory not being searched. Maybe if this is fixed, user-level bin directory doesn’t have to be added to the PATH environment variable any more.

This problem can also be reproduced using Docker:

docker run --rm python:3 sh -c 'pip install --user notebook && python3 -m jupyter notebook'

[ivanov]

Hey there, I'm going through old issues and it seems to me that it makes sense to close this one. Thanks to @pfmoore, this was implemented back in #162 .

Thanks everyone and happy hacking! :bowtie: