VS Code crashes when running a cell that produces a lot of output

[ydmykr]

I am trying to print model summary in tensorflow and I think the model is large and it's crashing my notebook. The model is ResNet101.

The whole computer comes to a halt, memory usage goes up to 99% and VS Code crashes. I have 16 GB of ram, so I didn't think printing something large would actually eat all my ram. Also, because the kernel crashes, all the variables are lost like history = model.fit() which I need to fine-tune the model afterwards. Moreover, I need to print base_model summary in order to choose from which layer to fine-tune from.

Is there a way to print the summary in another way and save the entire notebook with the variables, so I can continue working? I have checkpoints for model weights, but I need to keep track of past epochs through history to resume training afterwards.

I will try using.py files, but I want to know if there is a way to solve this problem for jupyter.

[DonJayamanne]

Thanks for filing this issue and I'm sorry you are running into this.

• Is it VS Code or is it is the kernel that crashes?
• If you were to run this outside VS Code in classic jupyter via a terminal , do you experience the same problem?
• Can you run a cell at a time to check when exactly vscode crashes, that would help narrow down the issue

[ydmykr]

@DonJayamanne 1.Vscode crashed 2.It doesn't crash on classic jupyter 3.The cell that cause the problem is the one where I am trying to print model summary, It consumes 99% of my available memory and it will either crash VS Code or it will gray everything out

[DonJayamanne]

Thanks for your patience & I'm sorry about this issue, hopefully we can get to the bottom of this.

I'm assuming you are referring to the cell with the code base_model.summary Is it possible to try the following:

• Not print any output at all
• Run the first cell, then clear all output using the command or icon in the toolbar
• Run the next cell and clear all of the output.
• Similarly continue till you get to the problematic cell, again, comment out the code where you are printing the model summary.

I have a suspicion the output from the second cell is very large & that could be chewing up resources. Again, please ensure you clera all of the output after running each cell.

What I'd like to figure out is:

• Whether the memory usaage/crash is realted to the output or execution
• If we ensure the output is always cleared, and things work as expected (i.e. it does not crash), then we've narrowed down the issue to outputs causing increased memory usage and crashes vscode.

Finally

• Also, please could you share the output from the Jupyter output panel, use the command Jupyter: Show Output Run the first cell in the notebook and copy the logs and send the logs (please ensure you send the whole log). Shows informatino about versions, python used, and the like.
• Are you running the code against a local or remote kernel?

• memory usage goes up to 99% and VS Code crashes. I Please could you take a screenshot of this and upload that here, would like to check what vscode process is chewing up the memory. thanks

[mariosconsta]

I am the OP who posted this question on stack overflow and OP here created the issue for me, so I will answer your questions. So after making a new notebook to test again, the same thing happens: Sorry for the phone screenshot, computer froze. Also, please note that because the computer is getting so slow, it can't show me the exact usage number, but I guess it's gigabytes since its saying 99%

• Clearing outputs is not changing the outcome

I have a suspicion the output from the second cell is very large & that could be chewing up resources. Again, please ensure you clera all of the output after running each cell.

• I have put verbose = 2 for model.fit so that output is not large. But eitherway, the same thing happens on another notebook where I do not train any model and the only output is base_model.summary()

Similarly continue till you get to the problematic cell, again, comment out the code where you are printing the model summary

Yep the problematic cell is the one where I print the summary(). Same thing happens on a brand new and small notebook that I just import ResNet101 and try to print it's summary.

What I'd like to figure out is:

Whether the memory usaage/crash is realted to the output or execution

I think it's execution and not output. Because even if 15-20secs has passed since I pressed run on that cell, after it finishes it shows 0.5s and when I run that cell again, it is taking less time which I believe is because the variable is already saved and it doesn't need to compute it again.

If we ensure the output is always cleared, and things work as expected (i.e. it does not crash), then we've narrowed down the issue to outputs causing increased memory usage and crashes vscode.

Unfortunately, that's not the issue

Are you running the code against a local or remote kernel?

Local Kernel

Also, please could you share the output from the Jupyter output panel, use the command Jupyter: Show Output Run the first cell in the notebook and copy the logs and send the logs (please ensure you send the whole log). Shows informatinon about versions, python used, and the like.

logs.txt

[DonJayamanne]

Thanks for all of the information, please could you:

• Provide a very small sample code to replicate this at our end. You mentioned a small notebook, please provide that as a sample or something even smaller.
• Please provide all of the necessary python packages that need to be installed, if you are using conda, then even better as you could just provide the conda yaml file (do let us know the version of the python environment as well).
• If there are ay other files that we need, please could you provide those as well (feel free to email them to me, if you think they are they shoudl not be shared online, my email is on my github profile)

[mariosconsta]

Thanks for all of the information, please could you:

• Provide a very small sample code to replicate this at our end. You mentioned a small notebook, please provide that as a sample or something even smaller.
• Please provide all of the necessary python packages that need to be installed, if you are using conda, then even better as you could just provide the conda yaml file (do let us know the version of the python environment as well).
• If there are ay other files that we need, please could you provide those as well (feel free to email them to me, if you think they are they shoudl not be shared online, my email is on my github profile)

The zip folder should contain everything you need. The small demo notebook at around 100KB and a notepad with all my packages, you can run pip install with that notepad and it will install everything for you.

demo.zip

Also I am using venv, python version is 3.10.2, tensorflow version is the latest. If you can't reproduce it, I can take a video with my phone and show you step by step what is happening.

@DonJayamanne Let me know if you need anything else and I would be more than happy to assist

[DonJayamanne]

@mariosconsta I've tried running this at my end, both on a Mac and Windows machine and have no problems.

Can you try the following:

• Step 1
  • Run the notebook in Jupyter notebooks, save the notebook and upload that saved notebook here.

    notebook where I do not train any model and the only output is base_model.summary()

  • Run the above notebook in Jupyter notebooks as well , save the notebook and upload that saved notebook here.
• Step 2
  • Exit vscode completely
  • Open the above two notebooks and see if you have similar perf issues in vscode.
  • Check the memory usage
  • Note: At this point, you are NOT running anything in VS Code, just opening the notebooks

Basically I'd like to get the two notebooks.

[mariosconsta]

Sorry for the late reply, I will test this out ASAP and let you know.

[mariosconsta]

@DonJayamanne

So with VS Code closed. I opened jupyter notebook and run the cell that outputs model summary. The cell ran instantly without trouble. I opened VS Code to try it again and the same thing happened.

notebooks.zip

I will take a video with my phone as its happening and upload it here as well.

[mariosconsta]

@DonJayamanne I have uploaded 2 clips of the problem. The second clip is from my phone because I couldn't record normally. The first clip is showing jupyter running like a charm and the second one is VS Code consuming 100% of my RAM and then crashing.

There's also 2 screenshots on how VS Code looks after it recovers along with the error that sometimes pops up. The error doesn't pop up always.

https://drive.google.com/file/d/1GzOu8ZdX9CTZeN7guRcpZpYaWco5B445/view?usp=sharing

[DonJayamanne]

Will get a custom build of he extension shortly, so you can test with that Once again, thanks for your patience and sorry for the inconvenience caused.

[DonJayamanne]

Please could you:

• Install VS Code Insiders
• Install the attached vsix using the command Extnsions: Install from VSIX
• Reload VS Code insiders
• Try running the notebook with this version
• Note: You might not see any outputs in any cell, however you should still see the green tick to indicate whether a cell ran successfully or not, along with the execution count

Please let me know how this goes. ms-toolsai-jupyter-insiders.vsix.zip

[mariosconsta]

I will test it out first thing in the morning!

[mariosconsta]

@DonJayamanne Runs instantly as shown in the screenshot below. Like you said there's no output, but it does run like a charm and uses zero ram. Is there a way to get output from this version? Or should I continue using the version I currently am on.

P.S. I tried printing it as well to see if I get output but I didn't. The original code was just

base_model.summary()

Also these are my extensions:

Should I try switching to jupyter pre-release version on VS Code? (not insiders, the normal one)

EDIT: So using the pre-release version of jupyter did not fix the problem.

VS Code (main version not the one you sent) still uses 100% of my memory and after that, I get that visual bug. Like I said, this does not happen on insiders version and with the file you shared. Also it does not happen on stand alone jupyter notebook.

The cell might say 0.5s but it wasn't 0.5s, it took more than 15s to complete.

EDIT:

I am willing to do an AnyDesk session with you if you want to see it "live" if the recordings I sent you were not enough.

In conclusion, the version you sent works like a charm, the main version does not, this text is all over the place and I am sorry for that.

[mariosconsta]

Another issue is this:

Sometimes when it finishes without crashing, the output looks like that.

[DonJayamanne]

@mariosconsta Please could you try installing this VSIX as well and letting me know whether this still crashes. Thanks again for your help and patience ms-toolsai-jupyter-insiders.vsix (1).zip

[DonJayamanne]

I am willing to do an AnyDesk session with you if you want to see it "live" if the recordings I sent you were not enough.

Thanks, so far what we have going is working, i'm still trying to gather some more information before I start working on this issue.

[mariosconsta]

@DonJayamanne

Please could you try installing this VSIX as well and letting me know whether this still crashes.

Just like the last VSIX, works like a charm. The cell runs instantly without issues.

[mariosconsta]

Please could you try installing this VSIX as well and letting me know whether this still crashes.

Also like before, no output. Could this be the issue? The output? I did some testing and I was able to output large text without issue using a for loop.

You can check the recording here: demo2.zip

[DonJayamanne]

Internal note: Blocked on upstream issue https://github.com/microsoft/vscode/issues/146768

Also like before, no output. Could this be the issue? The output? I did some testing and I was able to output large text without issue using a for loop.

@mariosconsta I'm sorry, but I dont follow you. From what I understand with the VSIX i provided, it doesn't crash anymore, however hot output is displayed is that right.

Could this be the issue? The output? I

Currently the extension host seems to crash due to the size of the output generated.

[mariosconsta]

I'm sorry, but I dont follow you.

Both VSIX you provided do not crash but do not show any output either.

Currently the extension host seems to crash due to the size of the output generated.

I see, well let me know if you need anything from me and if there's any way to work around this problem.

[mariosconsta]

Currently the extension host seems to crash due to the size of the output generated.

@DonJayamanne by the way, if you saw the recording in the demo2.zip, I printed only the layers of the model using a for loop, which is a very long list of text, and it runs instantly.

[DonJayamanne]

@mariosconsta Please could you upload the dataset folder as well. That wasn't in the zip file you sent earlier. Optionally could you send the instructions (links) to download the necessary data. thanks

Quick question: Does VS Code crash when you open the RESNET101 file you sent?

[mariosconsta]

@DonJayamanne I can provide my dataset through WeTransfer because it has a lot of images. Can you provide me with an email? I downloaded it through Kaggle, but I made some changes to it. Uploading to Google Drive will take some time. If you don't mind I can start uploading it to Google Drive and send you the link once it is done.

Quick question: Does VS Code crash when you open the RESNET101 file you sent?

Nope, VS Code crashes when I run base_model.summary().

If I run model.summary() it does not crash because the output is not that big. This is the output when i run model.summary()

You can see that base_model is included in there (densenet121) but it does not expand on it. But if I run base_model.summary(), it will give me the summary of all 427 layers in that model. I know the file I sent you was with ResNet101, I tested this on multiple models, all produce the same outcome (crash).

Let me know if there's anything else I can do for you.

[DonJayamanne]

@mariosconsta Thanks a lot for your help and patience, I'm trying to resolve this issue in VS Code, and have made some progres (which is why the issue was accidentally closed via a PR). However I still believe the fix will bot be sufficient.

• Attached is a VSIX,
• Please install the latest VS Code Insider version and this VSIX
• Open your above problematic notebook that crashes VS Code
• Run just that ONE notebook (run all cells or what ever steps you use to crash VS Code)
• A couple of text file will be created next to the notebook of the form <notebook file name><number>.log
• Please upload the log files and the ipynb file you used

These log files contain all of the output generated and added to the notebook, hence should not contain any personal information, it would only contain the same information taht wouuld be stored in the in the ipynb files as output. If you notebook file is at C:\temp\one.ipynb, then the logs files would be C:\temp\one.ipynb1.log, C:\temp\one.ipynb2.log & so on. One file for each cell.

If you are going to edit the log files, please let me know so that I can account for those.

I'd really appreciate it if you could provide these logs, that would help me replicate this at my end without any of the additonal packages. ms-toolsai-jupyter-insiders.vsix.zip

Thanks again

[mariosconsta]

Hey @DonJayamanne, I updated VS Code Insider and installed the attached VSIX file. I run all the code cells that produced the crashed. The logs are attached below.

Also, I could not find your email in your GitHub profile to send you the Dataset that you asked for. Can you send it to me if you still want the data?

ResNet101_logs.zip

I greatly appreciate your relentless efforts to solve this issue.

[DonJayamanne]

your GitHub profile to send you the Dataset tha

. First name = Don Last Name = Jayamanne

Thanks a lot, will check the logs

[mariosconsta]

@DonJayamanne Sending you the dataset through WeTransfer right now.

[DonJayamanne]

@mariosconsta I'm sorry we weren't able to get this fixed in the current iteration, we've been working on this issue, however it ended up taking a bit longer than expected (due to changes in the API and fine turning the code). Our plan is to get this addressed in the next release.

[mariosconsta]

@DonJayamanne No worries, let me know if you need anything else.

[codeananda]

Just read through this whole thread. I've experienced the same issue when running a long bit of code outputting plots overnight.

It crashed last night (indeed, it has crashed every time I've tried to run it).

Now two notebooks seem to be corrupted. One I was running last night (notebook 1) and the other (notebook 2) which also contains a lot of plots. I was working in notebook 2 yesterday but it was closed during the overnight crash. White rendering over all markdown cells and outputs cannot be seen. If you create a new code cell and execute, it doesn't produce any output.

Is there any way to recover the notebooks? I've tried opening them on VS Code Insiders but they are also broken there. However, when opening in native jupyter, the output of notebook 1 for the cell that caused the crash is gone. For notebook 2, all the plots are still there (which is good news!)

Update (next day): my boss sent me a notebook he worked on that contains similar code to notebook 2. He expressed no issues about it (he uses PyCharm). When I tired to open it in VS Code it had the White Walls of Doom™ and all the outputs were gone. Is this an issue with notebooks that contain a large amount of output in general?

[mariosconsta]

Yeah I can verify as well that it does not work. I run my notebook again and the whole computer froze for 3 solid minutes and then:

[DonJayamanne]

hi everyone which, but this should be fixed in the next release (we were unable to take last month).

you can try installing the latest pre-release version of the jupyter extension and that should work.

[codeananda]

I can confirm that the White Walls of Doom™ have gone and I am able to work with smaller notebooks.

However, I'm now unable to open some of the larger notebooks my colleague and I have been using. I'm accessing them withWSL and they are stored on Google Drive. He edited them and I want to open them. It tries to load for ages then gives this error:

It then closed another notebook I was working on and failed to save.

Are these errors related?

[dokutoshi]

The issue remains in the latest VScode. I am running a long Resnet mode to a remote server using VScode. Vscode crashed twice. VScode Jupyter is a white wall as described earlier. Just to remind you, I am not running the code locally but remotely.

[codeananda]

Update: I haven't had the white walls for a while. I've switched to putting all my files on the WSL filesystem and have been using a docker container as my development environment. I've also moved away from using notebooks as much and focused on coding in py files and using notebooks more like scratchpads to check bits of code work or to inspect plots.

Switching all files to WSL had a massive boost on performance in general.

Not sure if that's helpful but it's what has been working for me for time being.

Update a few hours later: I just tried to do a big calculation in a notebook and the White Walls of Doom™ came back. It seems like VS Code + Notebooks + Big Calculations = Bad Idea.

I will keep my notebook workload to a minimum moving forwards.

[mariosconsta]

@codeananda I have also stopped using notebooks for heavy workloads. I have one or two notebooks just for testing out individual code and then I implement it using regular python files. I did not try using docker and WSL but I will give it a go!

[DonJayamanne]

@mariosconsta @codeananda @dokutoshi I'm sorry you are still running into this issue.

• Assume you save the notebook and open it again in VS Code, does VS Code then crash again?
• Assume you run this notebook in regular Jupyter Notebook/Lab, and then open the notebook in VS Code, does it crash again

Basically trying to figure out whether its caused by

• The regular streaming of data
• Or because the output in a cell is too large

Please could you help us repro this issue

• Could you perhaps create a docker file with the necessary dependencies with instruction so we can test this at our end
• Please remember to include the same code (notebook) with instructions for installing all dependencies

just tried to do a big calculation in a notebook and the White Walls of Doom™ came back. It seems like VS Code + Notebooks + Big Calculations = Bad Idea.

@codeananda Is this in docker or WSL. Please could you share some sample code

Basically I'm looking for some sample code that would generate such large content to cause this issue at our end

[dokutoshi]

Will respond today.

For now, enumerations i.e. printing accuracy definitely affect memory.
Here is a screenshot of a G4dn AWS EC2 running an inference on 75,000 images.

The server is utilizing 53GB of RAM mostly because of VSCode server's nodeJS.
The MacBook Pro M1, 16GB RAM uses about 2GB to 10GB to manage the Remote SSH.

As a suggestion, ResNetv152 on a MNIST may be a good model. My model is a CV object detection on image data using bounding boxes. It's company data so cannot share the exact description.

[DonJayamanne]

Thanks, hopefully you can share something that can be run locally. Mac M1 is also fine

[tyePhDCandy]

same problem here.

vscode occupies like 45% of my cpu computing power. my cpu is amd 5700g and i have 64GB of memory.

I think the problem lies in the way of vscode-jupyter handles the output info when they are thousands of them.

i dont think the occassion is very easy to replicate, because it only happens on one of my notebook file.

I will report more, if i have the time.

[livan3li]

hi, has this problem been solved? @DonJayamanne

I have same problem right now. I have 24 GB ram and have recorded a video while problem is happening. VS Code consumes all available memory and OS crashes if i dont close the VS CODE When i try to open a Jupyter Notebook which have 618 images in.

But weird thing is that when i try to open same notebook in Google Colab everything is perfect. No memory consumption and even more i can view all images very fast by sliding scroll bar very very fast and as a result no freeze, no stuck, no problem.

I think the problem might be all about loading all data into memory(or memory management), not from storage device(HDD/SSD). Question is how Google Colab can achive such a good performance.

Versions: VS CODE Version: 1.76.2 (user setup) Commit: ee2b180d582a7f601fa6ecfdad8d9fd269ab1884 Date: 2023-03-14T17:55:54.936Z Electron: 19.1.11 Chromium: 102.0.5005.196 Node.js: 16.14.2 V8: 10.2.154.26-electron.0 OS: Windows_NT x64 10.0.19045 Sandboxed: Yes

Jupyter v2023.2.1200692131

Jupyter Keymap v1.1.0

Jupyter Notebook Renderers v1.0.15

Jupyter Slide Show v0.1.5

Thanks.

https://user-images.githubusercontent.com/46044710/227801773-ef050d9e-c9f9-4bd8-bf6c-f438184d5b45.mp4

[DonJayamanne]

@livan3li
Unfortunately this hasn't been fixed yet please could you share this notebook I'd like to try this at my end

Also, please could your disable all extensions, including Jupyter and see if you run into this issue If you do then the problem is different, as this issue focuses on crashes doe to running of notebooks However I do want to address your issue too, but would like to deal with that separately, for now let's chat here

[livan3li]

Ok. I've disabled all extensions related with Jupyter after this it's been opened as plain text file.

Here is the notebook link in drive which i have used in the video.

@livan3li Unfortunately this hasn't been fixed yet please could you share this notebook I'd like to try this at my end

Also, please could your disable all extensions, including Jupyter and see if you run into this issue If you do then the problem is different, as this issue focuses on crashes doe to running of notebooks However I do want to address your issue too, but would like to deal with that separately, for now let's chat here

[DonJayamanne]

I've disabled all extensions related with Jupyter after this it's been opened as plain text file.

What i meant was, disable all extensions (including Jupyter), but still open as a notebook. You can still open the ipynb file as a notebook even with teh Jupyter extension disabled, Please try to re-open the file as a notebook (instead of opening it as a plain text file) Opening and rendering notebooks is a feature that is a core functionality in vscode

[DonJayamanne]

@rebornix aren't you already looking into some perf improvements for opening notebooks with large number of outputs?

[livan3li]

I've disabled all extensions related with Jupyter after this it's been opened as plain text file.

What i meant was, disable all extensions (including Jupyter), but still open as a notebook. You can still open the ipynb file as a notebook even with teh Jupyter extension disabled, Please try to re-open the file as a notebook (instead of opening it as a plain text file) Opening and rendering notebooks is a feature that is a core functionality in vscode

Sorry for that. I've done exactly what you said. I disabled all extensions (even Python) and tried to open the notebook but same thing happened again. VS Code consumes all available memory.

[tyePhDCandy]

I've disabled all extensions related with Jupyter after this it's been opened as plain text file.

What i meant was, disable all extensions (including Jupyter), but still open as a notebook. You can still open the ipynb file as a notebook even with teh Jupyter extension disabled, Please try to re-open the file as a notebook (instead of opening it as a plain text file) Opening and rendering notebooks is a feature that is a core functionality in vscode

Sorry for that. I've done exactly what you said. I disabled all extensions (even Python) and tried to open the notebook but same thing happened again. VS Code consumes all available memory.

you can check the subprocess under main process 'vscode' . see which subprocess consumes all that memory. I found out, on my end, the python subprocess under 'vscode' consumes loads of memory.

[tyePhDCandy]

Yeah I can verify as well that it does not work. I run my notebook again and the whole computer froze for 3 solid minutes and then:

this happens to me too.

[livan3li]

I've disabled all extensions related with Jupyter after this it's been opened as plain text file.

What i meant was, disable all extensions (including Jupyter), but still open as a notebook. You can still open the ipynb file as a notebook even with teh Jupyter extension disabled, Please try to re-open the file as a notebook (instead of opening it as a plain text file) Opening and rendering notebooks is a feature that is a core functionality in vscode

Sorry for that. I've done exactly what you said. I disabled all extensions (even Python) and tried to open the notebook but same thing happened again. VS Code consumes all available memory.

you can check the subprocess under main process 'vscode' . see which subprocess consumes all that memory. I found out, on my end, the python subprocess under 'vscode' consumes loads of memory.

Since i disabled all extensions, Python doesn't do that for me. But now under VS Code, again VS Code consumes all memory.