search

microsoft / vscode-jupyter

VS Code Jupyter extension
https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter
MIT License
1.29k stars 290 forks source link

Disposing session as kernel process died ExitCode: undefined #13678

Closed abhyuditjain closed 1 year ago

abhyuditjain commented 1 year ago

Applies To

What happened?

I have a notebook that works fine on Google Colab. So I created an environment using pyenv with the same python version and package versions on my local system.

I have selected the interpreter as the correct one (I only have 1 virtual environment). After a few cells (where I import and define functions) run fine, but where I actually do some work, kernel crashes.

image

Packages

anyio==3.7.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.2.1
attrs==23.1.0
backcall==0.2.0
beautifulsoup4==4.12.2
bleach==6.0.0
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
cmake==3.26.4
comm==0.1.3
contourpy==1.0.7
cycler==0.11.0
debugpy==1.6.7
decorator==5.1.1
defusedxml==0.7.1
exceptiongroup==1.1.1
executing==1.2.0
fastjsonschema==2.17.1
filelock==3.12.1
fonttools==4.39.4
idna==3.4
ipykernel==6.23.1
ipython==8.14.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
jedi==0.18.2
Jinja2==3.1.2
jsonschema==4.17.3
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter-server==1.24.0
jupyter_core==5.3.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.7
kiwisolver==1.4.4
lit==16.0.5.post0
MarkupSafe==2.1.3
matplotlib==3.7.1
matplotlib-inline==0.1.6
mistune==2.0.5
mpmath==1.3.0
nbclassic==1.0.0
nbclient==0.8.0
nbconvert==7.4.0
nbformat==5.9.0
nest-asyncio==1.5.6
networkx==3.1
notebook==6.5.4
notebook_shim==0.2.3
numpy==1.24.3
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
packaging==23.1
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.5.0
platformdirs==3.5.3
prometheus-client==0.17.0
prompt-toolkit==3.0.38
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
Pygments==2.15.1
pyparsing==3.0.9
pyrsistent==0.19.3
python-dateutil==2.8.2
pyzmq==23.2.1
requests==2.31.0
Send2Trash==1.8.2
six==1.16.0
sniffio==1.3.0
soupsieve==2.4.1
stack-data==0.6.2
sympy==1.12
terminado==0.17.1
tinycss2==1.2.1
torch==2.0.1
torchinfo==1.8.0
torchvision==0.15.2
tornado==6.3.2
tqdm==4.65.0
traitlets==5.9.0
triton==2.0.0
typing_extensions==4.6.3
urllib3==2.0.3
wcwidth==0.2.6
webencodings==0.5.1
websocket-client==1.5.3
widgetsnbextension==3.6.4

VS Code Version

Version: 1.79.0 (user setup) Commit: b380da4ef1ee00e224a15c1d4d9793e27c2b6302 Date: 2023-06-07T14:26:35.552Z Electron: 22.5.5 Chromium: 108.0.5359.215 Node.js: 16.17.1 V8: 10.8.168.25-electron.0 OS: Windows_NT x64 10.0.22621

Jupyter Extension Version

v2023.5.1001582324

Jupyter logs

Visual Studio Code (1.79.0, wsl, desktop)
Jupyter Extension Version: 2023.5.1001582324.
Python Extension Version: 2023.10.0.
Platform: linux (x64).
Workspace folder ~/code/era-v1, Home = /home/carb0n
17:16:30.765 [info] Start refreshing Interpreter Kernel Picker (1686397590765)
17:16:30.797 [info] Using Pylance
17:16:32.526 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd092f6b483f4f356188c8ae93938b0ef11e2424c7fd732df3a4c589b7ce745b72a.~/.pyenv/versions/3.10.12/envs/era-v1-colab/python.~/.pyenv/versions/3.10.12/envs/era-v1-colab/python.-m#ipykernel_launcher  (Python Path: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python, Pyenv, era-v1-colab, 3.10.12) for '~/code/era-v1/session_04/Session_4.ipynb' (disableUI=true)
17:16:32.668 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python -m pip list
17:16:32.688 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
17:16:32.696 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"58b793e7-05d3-4aa7-ae2e-c47e40f7879d" --shell=9002 --transport="tcp" --iopub=9004 --f=~/.local/share/jupyter/runtime/kernel-v2-15985LpMGGKM6FnCr.json
    > cwd: ~/code/era-v1/session_04
17:16:32.919 [info] ipykernel version & path 6.23.1, ~/.pyenv/versions/3.10.12/envs/era-v1-colab/lib/python3.10/site-packages/ipykernel/__init__.py for ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python
17:16:33.571 [info] Started Kernel era-v1-colab (Python 3.10.12) (pid: 16112)
17:16:33.571 [info] Started new session d8ce17c8-0708-40ce-8769-4744d0e3e5ad
17:16:33.651 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python ~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/pythonFiles/printJupyterDataDir.py
17:16:34.262 [info] End refreshing Interpreter Kernel Picker (1686397590765)
17:16:41.492 [warn] No interpreter for Pylance for Notebook URI "~/code/era-v1/session_06/S6 copy.ipynb"
17:16:48.664 [info] Handle Execution of Cells 0,1,2,3,4,5,6,7,8,9,10 for ~/code/era-v1/session_06/S6 copy.ipynb
17:16:48.669 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd092f6b483f4f356188c8ae93938b0ef11e2424c7fd732df3a4c589b7ce745b72a.~/.pyenv/versions/3.10.12/envs/era-v1-colab/python.~/.pyenv/versions/3.10.12/envs/era-v1-colab/python.-m#ipykernel_launcher  (Python Path: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python, Pyenv, era-v1-colab, 3.10.12) for '~/code/era-v1/session_06/S6 copy.ipynb' (disableUI=false)
17:16:48.677 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
17:16:48.695 [info] Process Execution: ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"0baf86a3-8192-431d-a3ae-69d8415068ac" --shell=9007 --transport="tcp" --iopub=9009 --f=~/.local/share/jupyter/runtime/kernel-v2-1598596Se5op973HH.json
    > cwd: ~/code/era-v1/session_06
17:16:48.861 [info] ipykernel version & path 6.23.1, ~/.pyenv/versions/3.10.12/envs/era-v1-colab/lib/python3.10/site-packages/ipykernel/__init__.py for ~/.pyenv/versions/3.10.12/envs/era-v1-colab/bin/python
17:16:49.309 [info] Started Kernel era-v1-colab (Python 3.10.12) (pid: 16558)
17:16:49.309 [info] Started new session c03b3dd1-9725-4b99-b03f-01386065bb3e
17:16:49.402 [info] Kernel acknowledged execution of cell 0 @ 1686397609395
17:16:49.409 [info] End cell 0 execution @ 1686397609405, started @ 1686397609395, elapsed time = 0.01s
17:16:49.416 [info] Kernel acknowledged execution of cell 1 @ 1686397609411
17:16:51.564 [info] End cell 1 execution @ 1686397611561, started @ 1686397609411, elapsed time = 2.15s
17:16:51.564 [warn] Cancel all remaining cells due to cancellation or failure in execution
17:16:51.564 [info] End cell 2 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 3 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 4 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 5 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 6 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 7 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.564 [info] End cell 8 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.565 [info] End cell 9 execution @ undefined, started @ undefined, elapsed time = 0s
17:16:51.565 [info] End cell 10 execution @ undefined, started @ undefined, elapsed time = 0s
17:17:37.349 [info] Handle Execution of Cells 0,1,2,3,4,5,6,7,8,9,10 for ~/code/era-v1/session_06/S6 copy.ipynb
17:17:37.365 [info] Kernel acknowledged execution of cell 0 @ 1686397657360
17:17:37.371 [info] End cell 0 execution @ 1686397657367, started @ 1686397657360, elapsed time = 0.007s
17:17:37.378 [info] Kernel acknowledged execution of cell 1 @ 1686397657373
17:17:37.388 [info] End cell 1 execution @ 1686397657385, started @ 1686397657373, elapsed time = 0.012s
17:17:37.394 [info] Kernel acknowledged execution of cell 2 @ 1686397657390
17:17:37.398 [info] End cell 2 execution @ 1686397657395, started @ 1686397657390, elapsed time = 0.005s
17:17:37.661 [info] Kernel acknowledged execution of cell 3 @ 1686397657400
17:17:37.667 [info] End cell 3 execution @ 1686397657663, started @ 1686397657400, elapsed time = 0.263s
17:17:37.672 [info] Kernel acknowledged execution of cell 4 @ 1686397657668
17:17:37.676 [info] End cell 4 execution @ 1686397657673, started @ 1686397657668, elapsed time = 0.005s
17:17:37.685 [info] Kernel acknowledged execution of cell 5 @ 1686397657678
17:17:37.937 [info] End cell 5 execution @ 1686397657934, started @ 1686397657678, elapsed time = 0.256s
17:17:37.943 [info] Kernel acknowledged execution of cell 6 @ 1686397657938
17:17:37.951 [info] End cell 6 execution @ 1686397657947, started @ 1686397657938, elapsed time = 0.009s
17:17:37.961 [info] Kernel acknowledged execution of cell 7 @ 1686397657953
17:17:37.967 [info] End cell 7 execution @ 1686397657962, started @ 1686397657953, elapsed time = 0.009s
17:17:37.970 [info] Kernel acknowledged execution of cell 8 @ 1686397657968
17:17:37.987 [info] End cell 8 execution @ 1686397657983, started @ 1686397657968, elapsed time = 0.015s
17:17:37.992 [info] Kernel acknowledged execution of cell 9 @ 1686397657988
17:17:37.996 [info] End cell 9 execution @ 1686397657993, started @ 1686397657988, elapsed time = 0.005s
17:17:38.005 [info] Kernel acknowledged execution of cell 10 @ 1686397657998
17:17:38.084 [error] Disposing session as kernel process died ExitCode: undefined, Reason: 
17:17:38.085 [info] Dispose Kernel process 16558.
17:17:38.085 [error] Raw kernel process exited code: undefined
17:17:38.088 [error] Error in waiting for cell to complete [Error: Canceled future for execute_request message before replies were done
    at t.KernelShellFutureHandler.dispose (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:32375)
    at ~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:51427
    at Map.forEach (<anonymous>)
    at y._clearKernelState (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:51412)
    at y.dispose (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:44894)
    at ~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:24:112498
    at ne (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:1586779)
    at cy.dispose (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:24:112474)
    at uy.dispose (~/.vscode-server/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:24:119757)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)]
17:17:38.088 [warn] Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
17:17:38.090 [info] End cell 10 execution @ 1686397658090, started @ 1686397657998, elapsed time = 0.092s
17:17:38.090 [warn] Cancel all remaining cells due to cancellation or failure in execution
17:17:38.176 [info] End cell 10 execution @ undefined, started @ undefined, elapsed time = 0s

Coding Language and Runtime Version

Python 3.10.12

Language Extension Version (if applicable)

v2023.10.0

Anaconda Version (if applicable)

No response

Running Jupyter locally or remotely?

Local

MustafaDragonborne commented 1 year ago

I keep getting a similar kind of error a lot these days. Sometimes a certain cell runs fine, then I wait a bit and then try to re-run it again and it takes alot of time ultimately crashing and me having to restart the Kernel. This also happens alot when I'm debugging.

MustafaDragonborne commented 1 year ago

[info] Starting debugging with mode 1 10:38:15.296 [error] Disposing session as kernel process died ExitCode: undefined, Reason: invalid command name "140084133119744idle_draw" while executing "140084133119744idle_draw" ("after" script)

10:38:15.296 [info] Dispose Kernel process 158945. 10:38:15.296 [error] Raw kernel process exited code: undefined 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.311 [error] KernelDebugAdapter::handleMessage failure: Error: Canceled future for debug_request message before replies were done 10:38:15.314 [error] Jupyter Extension (Error in onKernelStatusChanged, method:onKernelStatusChanged): [Error: CANNOT execute cell that has been REMOVED from notebook at e.ExtHostNotebookKernels._createNotebookCellExecution (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:101:102321) at Object.createNotebookCellExecution (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:101:98283) at im.create (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:179570) at im.getOrCreate (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:179443) at Sm (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:188549) at xm.endCellAndDisplayErrorsInCell (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:196581) at xm.onKernelStatusChanged (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:196185) at r.value (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:2:1587966) at b.invoke (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:145) at d.deliver (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:2121) at n.fire (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:1729) at tg. (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:24:40359) at b.invoke (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:145) at d.deliver (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:2121) at n.fire (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:1729) at t (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:17:215219) at b.invoke (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:145) at d.deliver (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:2121) at n.fire (/snap/code/130/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:63:1729) at ~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1001582324-linux-x64/out/extension.node.js:24:124850 at at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]

DonJayamanne commented 1 year ago

Thank you for filing this issue and sorry you are running into this. In the past we have found that users run into similar issues when dealing with torch, pytorch or tensorflow Basically incorrectly installed packages such as the above can cause the kernel to crash, Right now the kernel is crashing when running code,

Please can you try running some hello world code like print("Hello World") in a cell and verify that it works. If this works, then this is an indication that the packages are to blame and we can look into that separately (right now i'd like to narrow down the issue)

Please let me know how things go

abhyuditjain commented 1 year ago

Hi @DonJayamanne,

I ran print() and it does work fine. However, that was working fine as I mentioned in my description. One more issue that I am facing is with tqdm. It used to work fine a while back, but now, whenever a print statement follows a tqdm set_description, it starts printing in new lines.

image

DonJayamanne commented 1 year ago

to work fine a while back, but now, whenever a print statement follows a tqdm set_description, it starts printing in new lines.

That’s problem in vscode and not Jupyter extension https://github.com/microsoft/vscode/issues/184657

DonJayamanne commented 1 year ago

and it does work fine.

can you share the Jupyter extension logs when you run the print statement i would like to see the details of the python env in the logs

after that, please run the same notebook in the Jupyter notebook/lab application against the same python environment,

Let me know if it fails in Jupyter lab/notebook as well and share the logs from the terminal

MustafaDragonborne commented 1 year ago

For me, even print won't work now. I was running some cells and everything was working fine until suddenly, the cells stop running and just show the small "clock" symbol in the cell showing that it's running but not really executing itself. New cells also display the same behaviour. My log shows [info] Handle Execution of Cells 33 for mynotebook.ipynb It's just stuck there. When I tried to restart the kernel, it told me I need to setup Ipykernel even though I have it installed. Ipykernel version I have is 6.23.2

DonJayamanne commented 1 year ago

@MustafaDragonborne looks like you are running into a different issue Please could you create a separate issue so we can look into that separately

TomasDemeter commented 1 year ago

@DonJayamanne I am having a very similar issue running jupyter notebooks both in vs code and in the browser and for both Python and R scripts. most of the cells run fine but some crash kernels with 'ExitCode: undefined'. Have you managed to fix the issue?

DonJayamanne commented 1 year ago

most of the cells run fine but some crash kernels with 'ExitCode: undefined'. Have you managed to fix the issue?

Does this crash in the browser as well. If it does, can you sahre the logs from the Jupyter in the terminal

Have you managed to fix the issue?

No, is generally an indicatiohn of the fact that there is a problem in the installed package, hence its an issue with the user environment (ie. python environment)

See here for information requested https://github.com/microsoft/vscode-jupyter/issues/13678#issuecomment-1589069402

TomasDemeter commented 1 year ago

@DonJayamanne sorry cannot post the logs anymore. I tried reinstalling my entire system but nothing helped. In the end, I ended up upgrading my RAM and that solved the issue. Not sure how helpful that is though.