microsoft / vscode-jupyter

VS Code Jupyter extension
https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter
MIT License
1.27k stars 284 forks source link

Notebook too large to backup #15871

Open ibobak opened 1 month ago

ibobak commented 1 month ago

Type: Bug

Behaviour

When a jupyter notebook cell is being executed, the rendering of output happens with huge delay. During this delay I am getting these logs:

024-07-24 21:36:44.139 [error] Error: Notebook too large to backup
    at r.snapshot (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2223:59569)
    at async I.backup (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2223:48595)
    at async vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2204:6230
2024-07-24 21:45:12.596 [error] [12f] potential listener LEAK detected, having 180 listeners already. MOST frequent listener (6):: Error
    at c.create (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:88:57139)
    at h.q [as onDidChange] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:91:1100)
    at Object.P [as onWillAddFirstListener] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:790:67117)
    at u.q [as onDidChange] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:91:1280)
    at k.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2496:109971)
    at k.render (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2496:107439)
    at P.w (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2496:113169)
    at P.render (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2496:112729)
    at new $ (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2496:148874)
    at f.o (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:655:1608)
    at f.createInstance (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:655:1100)
    at X.renderElement (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:2684:28983)
    at _.renderElement (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:227:2257)
    at T.Z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:184:16137)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:184:15157
    at n.transact (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:180:8020)
    at T.Y (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:184:15030)
    at T.Y (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:51171)
    at T.ib (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:184:20194)
    at l.B (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:732)
    at l.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:802)
    at l.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:1018)
    at d.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:180:55517)
    at l.B (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:732)
    at l.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:949)
    at L.n (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:26768)
    at L.setScrollPositionNow (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:93:25766)
    at E.P (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:180:49852)
    at C.b (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:180:49668)
    at HTMLDivElement.<anonymous> (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:140:1173)
2024-07-24 21:45:27.927 [error] [12f] potential listener LEAK detected, having 178 listeners already. MOST frequent listener (2):: Error
    at c.create (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:88:57139)
    at h.q [as onDidChange] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:91:1100)
    at Object.P [as onWillAddFirstListener] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:790:67117)
    at u.q [as onDidChange] (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:91:1280)
    at d.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:45843)
    at d.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:45352)
    at d.didRenderCell (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:45531)
    at d.renderCell (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:36116)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:36853
    at y (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:36307)
    at N.d (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:1134:36845)
    at N.execute (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:139:32615)
    at ot (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:139:32888)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:139:33100

Steps to reproduce:

I don't know how to reproduce in a test project. This is a commercial project - I can't share it, a notebook has 200 cells in it. It has many matplotlib outputs, many prints, logs, etc.

Diagnostic data

Output for Python in the Output panel (ViewOutput, change the drop-down the upper-right of the Output panel to Python)

``` XXX ```

Extension version: 2024.10.0 VS Code version: Code 1.91.1 (f1e16e1e6214d7c44d078b1f0607b2388f29d729, 2024-07-09T22:08:12.169Z) OS version: Linux x64 6.4.6-060406-generic Modes:

User Settings

``` languageServer: "Pylance" ```

Installed Extensions |Extension Name|Extension Id|Version| |---|---|---| |.NET Install Tool|ms-dotnettools.vscode-dotnet-runtime|2.1.1| |autoDocstring - Python Docstring Generator|njpwerner.autodocstring|0.6.1| |autopep8|ms-python.autopep8|2024.0.0| |Black Formatter|ms-python.black-formatter|2024.2.0| |C#|ms-dotnettools.csharp|2.39.29| |C# Dev Kit|ms-dotnettools.csdevkit|1.8.14| |Checkpoints|micnil.vscode-checkpoints|1.3.3| |Code Runner|formulahendry.code-runner|0.12.2| |Conventional Commits|vivaxy.vscode-conventional-commits|1.25.0| |Data Wrangler|ms-toolsai.datawrangler|1.4.2| |Dev Containers|ms-vscode-remote.remote-containers|0.375.1| |Django|batisteo.vscode-django|1.15.0| |Docker|ms-azuretools.vscode-docker|1.29.1| |Docs View|bierner.docs-view|0.1.0| |Edit csv|janisdd.vscode-edit-csv|0.9.2| |Flake8|ms-python.flake8|2023.10.0| |Git Extension Pack|donjayamanne.git-extension-pack|0.1.3| |Git Extension Pack (GPack)|SeyyedKhandon.gpack|2.0.0| |Git File History|pomber.git-file-history|1.0.1| |Git History|donjayamanne.githistory|0.6.20| |Github Light Theme|Hyzeta.vscode-theme-github-light|7.14.2| |GitHub Plus Theme|thenikso.github-plus-theme|1.4.3| |GitHub Theme|GitHub.github-vscode-theme|6.3.4| |gitignore|codezombiech.gitignore|0.9.0| |GitLens — Git supercharged|eamodio.gitlens|15.2.0| |Grey Light+ Pro|elanzalaco.grey-light-plus-pro|0.0.1| |Inline SQL|qufiwefefwoyn.inline-sql-syntax|2.16.0| |IntelliCode|VisualStudioExptTeam.vscodeintellicode|1.3.1| |IntelliCode API Usage Examples|VisualStudioExptTeam.intellicode-api-usage-examples|0.2.8| |IntelliCode for C# Dev Kit|ms-dotnettools.vscodeintellicode-csharp|2.1.11| |JavaScript Debugger|ms-vscode.js-debug|1.91.0| |JavaScript Debugger Companion Extension|ms-vscode.js-debug-companion|1.1.2| |Jinja|wholroyd.jinja|0.0.8| |julia light|JuliaLight.julia-light|0.4.0| |Jupyter|ms-toolsai.jupyter|2024.6.0| |Jupyter Cell Tags|ms-toolsai.vscode-jupyter-cell-tags|0.1.9| |Jupyter Keymap|ms-toolsai.jupyter-keymap|1.1.2| |Jupyter Notebook Renderers|ms-toolsai.jupyter-renderers|1.0.18| |Jupyter PowerToys|ms-toolsai.vscode-jupyter-powertoys|0.1.1| |Jupyter Slide Show|ms-toolsai.vscode-jupyter-slideshow|0.1.6| |Jupyter Theme|sam-the-programmer.jupyter-theme|0.1.3| |JupyterLab Light Theme|MiguelCorralJr.jupyterlab-light-theme|0.1.1| |Light Pink Theme|mgwg.light-pink-theme|0.6.0| |Markdown All in One|yzhang.markdown-all-in-one|3.6.2| |Markdown Preview Enhanced|shd101wyy.markdown-preview-enhanced|0.8.13| |MZ Light Grey Theme|Malaz-YI.mz-light-grey|2.1.1| |Night Owl|sdras.night-owl|2.0.1| |Notepad++ color themes|ShayanAhmedKhan.notepadpp-color-theme|0.2.0| |Numbered Bookmarks|alefragnani.numbered-bookmarks|8.5.0| |One Dark Pro|zhuangtongfa.material-theme|3.17.2| |One Dark Theme|mskelton.one-dark-theme|1.14.2| |Open in GitHub, Bitbucket, Gitlab, VisualStudio.com !|ziyasal.vscode-open-in-github|1.3.6| |Path Intellisense|christian-kohler.path-intellisense|2.9.0| |Project Manager|alefragnani.project-manager|12.8.0| |Pylance|ms-python.vscode-pylance|2024.7.1| |Pylint|ms-python.pylint|2023.10.1| |Python|ms-python.python|2024.10.0| |Python Debugger|ms-python.debugpy|2024.8.0| |Python Environment Manager|donjayamanne.python-environment-manager|1.2.4| |Python Extension Pack|donjayamanne.python-extension-pack|1.7.0| |Python Extension Pack|LeoJhonSong.python-extension-pack|3.0.0| |Python Indent|KevinRose.vsc-python-indent|1.18.0| |Remote - SSH|ms-vscode-remote.remote-ssh|0.112.0| |Remote - SSH: Editing Configuration Files|ms-vscode-remote.remote-ssh-edit|0.86.0| |Remote - Tunnels|ms-vscode.remote-server|1.5.2| |Remote Development|ms-vscode-remote.vscode-remote-extensionpack|0.25.0| |Remote Explorer|ms-vscode.remote-explorer|0.4.3| |Studio Icons|jtlowe.vscode-icon-theme|1.6.6| |Table Visualizer for JavaScript Profiles|ms-vscode.vscode-js-profile-table|1.0.9| |Visual Studio Keymap|ms-vscode.vs-keybindings|0.2.1| |WSL|ms-vscode-remote.remote-wsl|0.88.2| |Zoomer|anthonyattard.zoomer|0.3.1|
System Info |Item|Value| |---|---| |CPUs|Intel(R) Xeon(R) CPU E5-2696 v4 @ 2.20GHz (88 x 1453)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: disabled_software
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: disabled_off| |Load (avg)|2, 2, 2| |Memory (System)|251.76GB (218.99GB free)| |Process Argv|--crash-reporter-id 27d42247-63fb-4d9e-9cb0-87d9974843dc| |Screen Reader|no| |VM|50%| |DESKTOP_SESSION|ubuntu-xorg| |XDG_CURRENT_DESKTOP|Unity| |XDG_SESSION_DESKTOP|ubuntu-xorg| |XDG_SESSION_TYPE|x11|
A/B Experiments ``` vsliv368:30146709 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscoreces:30445986 vscod805:30301674 binariesv615:30325510 vsaa593:30376534 py29gd2263:31024239 c4g48928:30535728 azure-dev_surveyone:30548225 2i9eh265:30646982 962ge761:30959799 pythongtdpath:30769146 welcomedialog:30910333 pythonnoceb:30805159 asynctok:30898717 pythonregdiag2:30936856 pythonmypyd1:30879173 h48ei257:31000450 pythontbext0:30879054 accentitlementst:30995554 dsvsc016:30899300 dsvsc017:30899301 dsvsc018:30899302 cppperfnew:31000557 dsvsc020:30976470 pythonait:31006305 dsvsc021:30996838 9c06g630:31013171 pythoncenvpt:31062603 a69g1124:31058053 dvdeprecation:31068756 dwnewjupyter:31046869 2f103344:31071589 impr_priority:31102340 refactorc:31101458 wkspc-onlycs-t:31102394 ```
guidocioni commented 1 month ago

This started happening to me as well.. My notebook does not have many cells but it has one cell with a plotly trace that contains many scatter points, which is causing the ipynb to hover around 95 MB in size. I understand this is large for a notebook, but I don't understand what this "backup" should be and in any case it shouldn't really make the saving process hang for a few seconds.

amunger commented 1 month ago

backing up happens to be able to restore unsaved content, so if the file is saved, no backup is needed. Failing to backup occurs specifically to avoid causing any performance issues, so that shouldn't be the cause of any delay. The other log about too many listeners could be causing issues tho

guidocioni commented 1 month ago

backing up happens to be able to restore unsaved content, so if the file is saved, no backup is needed. Failing to backup occurs specifically to avoid causing any performance issues, so that shouldn't be the cause of any delay. The other log about too many listeners could be causing issues tho

Well I cycled through all the output logs and couldn't find anything writing at the same time I was saving the file... The only one is the notebook backup.

But I have to say that the original issue was citing celle execution while for me the Problem is indeed a delay in saving this particular notebook.

amunger commented 1 month ago

Saving a notebook that large can indeed put some pressure on performance - the entire 90 MB JSON string is serialized before it's written to disk.