Closed justmars closed 1 year ago
With the release version (1.74.3), the notebook works fine:
mkdir nb && cd ipynb
pyenv local 3.11.1
python -m venv .venv
source .venv/bin/activate
pip install jupyter
touch nb.ipynb
# open notebook in vscode, check proper kernel is selected, run import sys in cell, works
With the insiders version 1.75.0-insider, the notebook now shows a different error
Failed to start the Kernel. Jupyter server crashed. Unable to connect. Error code from Jupyter: 127 pyenv: jupyter-notebook: command not found
The `jupyter-notebook' command exists in these Python versions:
3.11.1
Note: See 'pyenv help global' for tips on allowing both python2 and python3 to be found. View Jupyter log for further details.
Here are the logs in case helpful:
With the release version (1.74.3), the notebook works fine: pip install jupyter
Please could you try installing as follows
python -m pip install jupyter ntoebook -U --force-reinstall
Jupyter server crashed. Unable to connect. Error code from Jupyter: 127 pyenv: jupyter-notebook: command not found The `jupyter-notebook' command exists in these Python versions: 3.11.1
Agreed, this seems to be related to some Python Path issues. We have cleaned up some code and stopped making assumptions about some path variables as it seems that things worked most likey by chance at this stage.
Please could you use the latest insiders and then Please could you enable logging as follows:
Jupyter->Logging
verbose
Jupyter
output panel.Weirdly, when trying the same command (see above getJupyterKernelspecVersion.py) in the terminal, it works fine: ~/Code/nb/.venv/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2023.1.2000131034/pytho
Can you type the following in the terminal after you run python -m pip install jupyter ntoebook -U --force-reinstall
which python
which jupyter
VSCode: 1.75.0-insider Python Extension: v2023.1.10111008 Jupyter Extension: v2023.1.2000141019
pyenv local 3.11.1
python -m venv .venv
source .venv/bin/activate
python -m pip install jupyter notebook -U --force-reinstall
touch nb.ipynb
# run import sys on first cell of notebook in Jupyter, error; see logs
which python
# /Users/mv/Code/nb/.venv/bin/python
which jupyter
# /Users/mv/.pyenv/shims/jupyter
python -c "import jupyter;print(jupyter.file)"
# AttributeError: module 'jupyter' has no attribute 'file'
Oops, i meant the following
python -c "import jupyter;print(jupyter.__file__)"
Thanks for providing the information requested and thanks for your patience.
which jupyter
/Users/mv/.pyenv/shims/jupyter
Yup, this seems to be the problem. for some reason jupyter is not getting installed into your virtual environment, it only gets installed in your pyenv directory.
pyenv local 3.11.1
Do you know how or why this is getting printed in the terminal. This explains a few things, it seems pyenv is getting initialized in your shell, but not in VS Code (VS Code doesn't have any knowledge of pyenv, only uses the virtual env .venv
in this case).
pyenv local 3.11.1
which python
Jupyter: Select Interpreter to Start Jupyter Server
Oops, i meant the following python -c "import jupyter;print(jupyter.file)"
python -c "import jupyter;print(jupyter.__file__)"
# /Users/mv/Code/nb/.venv/lib/python3.11/site-packages/jupyter.py
Do you know how or why this is getting printed in the terminal. This explains a few things, it seems pyenv is getting initialized in your shell, but not in VS Code (VS Code doesn't have any knowledge of pyenv, only uses the virtual env .venv in this case).
With respect to pyenv, python, and jupyter, these are the relevant environment variables I have in .zshrc
:
export JUPYTER_PLATFORM_DIRS=1 # see Jupyter DeprecationWarning
export PYTHONDEVMODE=1 # see https://docs.python.org/3.12/library/devmode.html
export PYENV_ROOT="$HOME/.pyenv" # pyenv python version
...
export PATH="$PYENV_ROOT:$POETRY_ROOT:$SQLITE:$PATH"
...
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi
The resulting environment (parameters above):
PATH=/Users/mv/Code/nb/.venv/bin:/Users/mv/.pyenv/shims:/Users/mv/.pyenv:/Users/mv/.local/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Applications/Postgres.app/Contents/Versions/latest/bin
JUPYTER_PLATFORM_DIRS=1
PYENV_ROOT=/Users/mv/.pyenv
POETRY_ROOT=/Users/mv/.local/bin
PYENV_SHELL=zsh
VIRTUAL_ENV=/Users/mv/Code/nb/.venv
Is this because pyenv init -
initializes the .venv
's pyvenv.cfg
to the following output:
home = /Users/mv/.pyenv/versions/3.11.1/bin
include-system-site-packages = false
version = 3.11.1
executable = /Users/mv/.pyenv/versions/3.11.1/bin/python3.11
command = /Users/mv/.pyenv/versions/3.11.1/bin/python3 -m venv /Users/mv/Code/nb-test/.venv
... and the command
portion enables initialization of python's.venv
... but no such command is applied to jupyter?
- Could you open a new terminal and run the following command
which python
which python
# /Users/mv/Code/nb/.venv/bin/python
- If that still fails, please send the same logs that you sent earlier (please ensure to send all of the logs, the initial bit was missing in the first log you sent)
and the command portion enables initialization of python's.venv... but no such command is applied to jupyter? Yes and the path variables are slightly different.
Please could you
.env
file in your /Users/mv/Code/nb
directoryPATH=/Users/mv/Code/nb/.venv/bin:/Users/mv/.pyenv/shims:/Users/mv/.pyenv:/Users/mv/.local/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Applications/Postgres.app/Contents/Versions/latest/bin
JUPYTER_PLATFORM_DIRS=1
PYENV_ROOT=/Users/mv/.pyenv
POETRY_ROOT=/Users/mv/.local/bin
PYENV_SHELL=zsh
VIRTUAL_ENV=/Users/mv/Code/nb/.venv
Next
/Users/mv/Code/nb/.venv/bin/python -m jupyter notebook --notebook-dir="/Users/mv/Code/nb"
.venv
Python environmentPython
import sys import os print(sys.path) print(os.getenv("PYTHONPATH"))
.env
file, cell errors out..env
file, running the cell works.The command /Users/mv/Code/nb/.venv/bin/python -m jupyter notebook --notebook-dir="/Users/mv/Code/nb"
works, resulting in opening a browser tab at http://localhost:8889/tree
. Exiting this and the terminal, then running the requested commands in an .venv activated python shell produces the following:
(.venv) ➜ nb python
Python 3.11.1 (main, Dec 9 2022, 22:35:02) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import os
>>> print(sys.path)
['', '/Users/mv/.pyenv/versions/3.11.1/lib/python311.zip', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11/lib-dynload', '/Users/mv/Code/nb/.venv/lib/python3.11/site-packages']
>>> print(os.getenv("PYTHONPATH"))
None
Ok, the problem is to do with PyEnv and virtualenvs created with PyEnv, the paths are a little funny and the packages are not getting installed into the isolated virtual env, but require the paths to pyven and loads jupyter from there.
As you can see here
['', '/Users/mv/.pyenv/versions/3.11.1/lib/python311.zip', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11/lib-dynload'
Python will look for the libraries first in pyenv folders, and thats why it works for you.
For now I would suggest you add the paths into the .env
file, you do not need to add all the paths, just the paths for pyenv
and you can use a syntax as follows
PATH=/Users/mv/.pyenv/shims:/Users/mv/.pyenv:$PATH
Check if that works, basically you can avoid hardcoding everything in the .env
file
In the mean time, i'll see what I can do to get a more generic fix.
Thanks for providing all of the information, please could you provie the following as well
.venv
environment
Run the following commands and let me know what you get for each of them
which python
python -m pip list
.vent
environment and run the same and let me know what you get (interested in the packages)
which python
python -m pip list
Thanks for providing all of the information, please could you provie the following as well
- Open a terminal (in a different folder without the
.venv
environment Run the following commands and let me know what you get for each of themwhich python python -m pip list
- Next open the terminal and activate the
.vent
environment and run the same and let me know what you get (interested in the packages)which python python -m pip list
In a separate folder Code
with no venv deactivated:
➜ Code which python
/Users/mv/.pyenv/shims/python
➜ Code python -m pip list
Package Version
----------------- -------
appnope 0.1.3
asttokens 2.1.0
backcall 0.2.0
debugpy 1.6.3
decorator 5.1.1
entrypoints 0.4
executing 1.2.0
ipykernel 6.17.0
ipython 8.6.0
jedi 0.18.1
jupyter_client 7.4.4
jupyter_core 4.11.2
matplotlib-inline 0.1.6
nest-asyncio 1.5.6
packaging 21.3
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 22.3.1
prompt-toolkit 3.0.32
psutil 5.9.3
ptyprocess 0.7.0
pure-eval 0.2.2
Pygments 2.13.0
pyparsing 3.0.9
python-dateutil 2.8.2
pyzmq 24.0.1
setuptools 65.5.0
six 1.16.0
stack-data 0.6.0
tornado 6.2
traitlets 5.5.0
wcwidth 0.2.5
In the Code/nb
folder with .venv activated:
➜ Code cd nb
➜ nb source .venv/bin/activate
(.venv) ➜ nb which python
/Users/mv/Code/nb/.venv/bin/python
(.venv) ➜ nb python -m pip list
Package Version
------------------------ -----------
anyio 3.6.2
appnope 0.1.3
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
asttokens 2.2.1
attrs 22.2.0
backcall 0.2.0
beautifulsoup4 4.11.1
bleach 5.0.1
cffi 1.15.1
comm 0.1.2
debugpy 1.6.5
decorator 5.1.1
defusedxml 0.7.1
entrypoints 0.4
executing 1.2.0
fastjsonschema 2.16.2
fqdn 1.5.1
idna 3.4
ipykernel 6.20.2
ipython 8.8.0
ipython-genutils 0.2.0
ipywidgets 8.0.4
isoduration 20.11.0
jedi 0.18.2
Jinja2 3.1.2
jsonpointer 2.3
jsonschema 4.17.3
jupyter 1.0.0
jupyter_client 7.4.9
jupyter-console 6.4.4
jupyter_core 5.1.3
jupyter-events 0.6.3
jupyter_server 2.1.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments 0.2.2
jupyterlab-widgets 3.0.5
MarkupSafe 2.1.1
matplotlib-inline 0.1.6
mistune 2.0.4
nbclassic 0.4.8
nbclient 0.7.2
nbconvert 7.2.8
nbformat 5.7.3
nest-asyncio 1.5.6
notebook 6.5.2
notebook_shim 0.2.2
packaging 23.0
pandocfilters 1.5.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 22.3.1
platformdirs 2.6.2
prometheus-client 0.15.0
prompt-toolkit 3.0.36
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pycparser 2.21
Pygments 2.14.0
pyrsistent 0.19.3
python-dateutil 2.8.2
python-json-logger 2.0.4
PyYAML 6.0
pyzmq 25.0.0
qtconsole 5.4.0
QtPy 2.3.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
Send2Trash 1.8.0
setuptools 65.5.0
six 1.16.0
sniffio 1.3.0
soupsieve 2.3.2.post1
stack-data 0.6.2
terminado 0.17.1
tinycss2 1.2.1
tornado 6.2
traitlets 5.8.1
uri-template 1.2.0
wcwidth 0.2.6
webcolors 1.12
webencodings 0.5.1
websocket-client 1.4.2
widgetsnbextension 4.0.5
Ok, the problem is to do with PyEnv and virtualenvs created with PyEnv, the paths are a little funny and the packages are not getting installed into the isolated virtual env, but require the paths to pyven and loads jupyter from there.
As you can see here
['', '/Users/mv/.pyenv/versions/3.11.1/lib/python311.zip', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11', '/Users/mv/.pyenv/versions/3.11.1/lib/python3.11/lib-dynload'
Python will look for the libraries first in pyenv folders, and thats why it works for you. For now I would suggest you add the paths into the
.env
file, you do not need to add all the paths, just the paths forpyenv
and you can use a syntax as followsPATH=/Users/mv/.pyenv/shims:/Users/mv/.pyenv:$PATH
Check if that works, basically you can avoid hardcoding everything in the
.env
file In the mean time, i'll see what I can do to get a more generic fix.
Just adding this line in the .env works:
PATH=/Users/mv/Code/nb/.venv/bin:/Users/mv/.pyenv/shims:/Users/mv/.pyenv:$PATH
If just the pyenv files without the initial .venv, it fails:
PATH=/Users/mv/.pyenv/shims:/Users/mv/.pyenv:$PATH
I also face the same problem, but .env
file does not fix it. Maybe because I use pyenv create miniconda environment?
@justmars This would indicate the fact that /Users/mv/.pyenv/shims
is not in your current path
I'll see what we can do to address that at our end.
@tcztzy
Jupyter->Logging
verbose
Jupyter
output panel.@tcztzy
- Open settings & go into
Jupyter->Logging
- Change the value to
verbose
- Reload VS Code,
- Attempt to repro the issue & then copy the output from the
Jupyter
output panel.
Fixed by reinstall all the virtual environment and clean python extension's cache.
Recent updates apparently break the present .env fix described above:
mkdir nb-test && code nb-test
pyenv local 3.11.2
python -m venv .venv
source .venv/bin/activate
python -m pip install ipykernel notebook jupyter
Add to env
PATH=/Users/mv/Code/nb-test/.venv/bin:/Users/mv/.pyenv/shims:/Users/mv/.pyenv:$PATH
Environment:
Notebook
import sys
in the first cell.venv (Python 3.11.2)
@justmars looks like you are running into an upstream issue see here https://github.com/jupyter/notebook/issues/6721 See here for a work around https://github.com/microsoft/vscode-jupyter/issues/12775#issuecomment-1447175314
Closing this as this should be fixed in the latest pre-release version of the Jupyter extesnsion.
Re: issue described under #12439
Hi @DonJayamanne , I tried with the new patch but now getting a kernelspec error, the modal prompt stating:
And in the notebook cell:
Environment
VS Code version: Version: 1.75.0-insider (Universal Jupyter Extension version: v2023.1.2000131034 Python Extension version: v2023.1.10111008 OS (Windows | Mac | Linux distro) and version: Mac Python and/or Anaconda version: Python 3.11.1 Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv Jupyter server running: ?
Steps to reproduce
In case the path ENV is helpful:
Jupyter Output Logs
Jupyter Output
```console debug 15:15:25.923: VSCodeNotebookController::handleExecution, Class name = Ov (started execution) debug 15:15:25.924: Execute Cell 0 /Users/mv/Code/nb/nb.ipynb debug 15:15:25.924: start the kernel, options.disableUI=false for /Users/mv/Code/nb/nb.ipynb debug 15:15:25.925: Start Jupyter Session in kernel.ts with disableUI = false info 15:15:25.925: Starting Jupyter Session startUsingPythonInterpreter, .jvsc74a57bd0cb7853ece70b40686b4af14f9f0121d44caa159afecbb8bc9ea6a753222ce8bd./Users/mv/Code/nb/.venv/python./Users/mv/Code/nb/.venv/python.-m#ipykernel_launcher (Python Path: /Users/mv/Code/nb/.venv, EnvType: Venv, EnvName: '.venv', Version: 3.11.1 (main, Dec 9 2022, 22:35:02) [Clang 14.0.0 (clang-1400.0.29.202)]) for '/Users/mv/Code/nb/nb.ipynb' (disableUI=false) info 15:15:25.925: Checking for server existence. debug 15:15:25.925: Checking for server usability. debug 15:15:25.926: Server not usable (should ask for install now) debug 15:15:25.926: Get Active Interpreter, Class name = Ss (started execution), Arg 1: undefined debug 15:15:25.926: Get Active Interpreter, Class name = Ss, completed in 0ms, has a truthy return value, Arg 1: undefined, Return Value: {"id":"/Users/mv/Code/nb/.venv/bin/python","sysPrefix":"/Users/mv/Code/nb/.venv","envPath":{"$mid":1,"path":"/Users/mv/Code/nb/.venv","scheme":"file"},"displayPath":{"$mid":1,"path":"/Users/mv/Code/nb/.venv","scheme":"file"},"envName":".venv","uri":{"$mid":1,"fsPath":"/Users/mv/Code/nb/.venv/bin/python","path":"/Users/mv/Code/nb/.venv/bin/python","scheme":"file"},"displayName":".venv","envType":"Venv","version":{"major":3,"minor":11,"patch":1,"raw":"3.11.1 (main, Dec 9 2022, 22:35:02) [Clang 14.0.0 (clang-1400.0.29.202)]"}} debug 15:15:25.926: Create activated Env, Class name = mc (started execution), Arg 1: {"interpreter":{"id":"/Users/mv/Code/nb/.venv/bin/python","sysPrefix":"/Users/mv/Code/nb/.venv","envPath":{"$mid":1,"path":"/Users/mv/Code/nb/.venv","scheme":"file"},"displayPath":{"$mid":1,"path":"/Users/mv/Code/nb/.venv","scheme":"file"},"envName":".venv","uri":{"$mid":1,"fsPath":"/Users/mv/Code/nb/.venv/bin/python","external":"file:///Users/mv/Code/nb/.venv/bin/python","path":"/Users/mv/Code/nb/.venv/bin/python","scheme":"file"},"displayName":".venv","envType":"Venv","version":{"major":3,"minor":11,"patch":1,"raw":"3.11.1 (main, Dec 9 2022, 22:35:02) [Clang 14.0.0 (clang-1400.0.29.202)]"}},"allowEnvironmentFetchExceptions":true} debug 15:15:25.926: Getting activated env variables, Class name = _s (started execution), Arg 1: "~/Code/nb", Arg 2:Notes on Logs
The logs until this part seems to work:
Working: ~/Code/nb/.venv/bin/python -m jupyter kernelspec --version
```console /Users/mv/Code/nb/.venv/lib/python3.11/site-packages/jupyter_core/command.py:20: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs given by the platformdirs library. To remove this warning and see the appropriate new directories, set the environment variable `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`. The use of platformdirs will be the default in `jupyter_core` v6 from . import paths /Users/mv/Code/nb/.venv/lib/python3.11/site-packages/jupyter_client/connect.py:27: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs given by the platformdirs library. To remove this warning and see the appropriate new directories, set the environment variable `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`. The use of platformdirs will be the default in `jupyter_core` v6 from jupyter_core.paths import jupyter_data_dir 7.4.9 ```Not working: ~/Code/nb/.venv/bin/python ~/.vscode-insiders/extensions/ms-toolsai.jupyter-2023.1.2000131034/pythonFiles/vscode_datascience_helpers/getJupyterKernelspecVersion.py
```console error 15:15:26.862: Failed to get a list of the kernelspec using python script [Ua [Error]: /Users/mv/Code/nb/.venv/lib/python3.11/site-packages/jupyter_client/connect.py:27: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs given by the platformdirs library. To remove this warning and see the appropriate new directories, set the environment variable `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`. The use of platformdirs will be the default in `jupyter_core` v6 from jupyter_core.paths import jupyter_data_dir at ChildProcess.Weirdly, when trying the same command (see above
getJupyterKernelspecVersion.py
) in the terminal, it works fine:Originally posted by @justmars in https://github.com/microsoft/vscode-jupyter/issues/12439#issuecomment-1379770996