Crash when running overnight

[Tyriar]

Ubuntu 12.04, VSCode 0.10.1

Several times VS Code has become unresponsive overnight on the above configuration (locked). Here is the program output:

$ code .
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

<--- Last few GCs --->

173527197 ms: Scavenge 1397.0 (1457.6) -> 1397.0 (1457.6) MB, 1.8 / 0 ms [allocation failure] [incremental marking delaying mark-sweep].
173527199 ms: Scavenge 1397.0 (1457.6) -> 1397.0 (1457.6) MB, 1.9 / 0 ms [allocation failure] [incremental marking delaying mark-sweep].
173529040 ms: Mark-sweep 1397.0 (1457.6) -> 1396.9 (1457.6) MB, 1841.7 / 98 ms [last resort gc].
173530775 ms: Mark-sweep 1396.9 (1457.6) -> 1396.1 (1457.6) MB, 1735.0 / 5 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x8a48933a859 <String[7]: file://>
    1: _completed [file:////home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/vs/workbench/workbench.main.js:~1544] [pc=0x23ff9b465433] (this=0x1a37262790b1 <JS Object>,e=0x1cd36e9041b9 <undefined>)
    2: arguments adaptor frame: 0->1
    6: bound  [native v8natives.js:1208] [pc=0x23ff99a26270] (this=0x8a489346089 <JS Global Object>)

==== Details =============================...

Failed to get crash dump id.
Report Id: 
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: channel closed
    at process.target.send (internal/child_process.js:509:16)
    at Console.console.error (/home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/bootstrap.js:5:937)
    at process.<anonymous> (/home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/bootstrap.js:5:1340)
    at emitOne (events.js:77:13)
    at process.emit (events.js:169:7)
    at process._fatalException (node.js:223:26)
[VS Code]: detected unresponsive

This has never occurred with Atom.

[Tyriar]

This seems to happen consistently every night I leave the computer running. It looks like it's unresponsive due to the CPU being maxed out. Maybe some infinite loop is somewhere, perhaps to do with file or git repo polling?

[bpasero]

I wonder if the OS decides to put VSCode (Electron actually) into a state where it is causing this crash. I have pinged Electron if they have a clue.

[Tyriar]

Never happened on Atom fyi, at least on 1.19.x (from memory) and 1.2.4.

[BobVul]

I have a similar issue, since the November update (0.10.2/0.10.3?). Just about every day I'd log in to find my VSCode windows left overnight have all crashed (with the standard uninformative/apologetic crash error, "Visual Studio Code has crashed").

Today, after the 0.10.5 update, I had my first crash while I was there - unfortunately not while I was actively using it.

Running VSCode on Windows 7 (64-bit), primarily using it as a JS editor on a very large project - nearly a million lines total (including libs I need to search, so are not excluded). No performance issues in normal use and I didn't notice any excessive resource usage leading up to today's crash.

I'd be happy to provide more detailed error logs/info if someone can point me to them.

[gwynjudd]

I'm getting the same basic issue as @Elusive138, when I leave code running overnight (every night), in the morning without fail I get "Visual Studio Code has crashed".

[Tyriar]

Still repros for me on vscode 0.10.5, Ubuntu 12.04

[bpasero]

I tried to reproduce on Windows 10, Mac OS 10.11 and Ubuntu 15 without luck. I am suspecting an out of memory issue but for none of the above the memory was increasing much.

Can someone try to reproduce this with the following conditions:

• does it reproduce when opening just an empty instance of code (File | Close Folder)
• does it reproduce when opening a workspace but not opening any file in the editor

[Tyriar]

Ubuntu 12.04, vscode 0.10.5 could not reproduce leaving it over the weekend with an empty instance.

[BobVul]

It may well be a subtle memory leak, made apparent by the relatively high memory utilisation on this system.

I left the computer at about 5:30 PM, with system memory commit slowly creeping up - it was 15,402 MB at 7:00 PM. At 3:09 AM was the closest approach to the system commit limit (17,682 MB), and commit usage dropped from 16,218 MB to 15,217 MB. I suspect this might be where VSCode crashed. Commit usage was stable around there until logging stopped around 6 AM (out of disk space - those performance counters are big!).

Unfortunately I did not include all the VSCode processes so I do not have process-specific logging. I will try that tonight.

Would be very useful if I could get the time of crash. Does VSCode leave logs anywhere?

[bpasero]

Unfortunately I did not include all the VSCode processes so I do not have process-specific logging. I will try that tonight.

That would be super useful, thanks!

Currently vscode does not log to disk.

[bpasero]

@Elusive138 can you share the workspace you let vscode running on?

[BobVul]

@bpasero Unfortunately not. It is based on Ext JS, though, and that's where the majority of the (library) code is. I'll try a clean Ext JS workspace after this other testing is done, and see if it repros there.

[bpasero]

@Elusive138 yes would be good to have a sample to reproduce on our end.

[BobVul]

One of the code.exe processes (code#3 in the log, attached) seems to be leaking. Commit started at about 200 MB at 5:30 PM, and reached 460 MB by 9:00 AM the next day, with a constant increase:

Handle count does not go up.

vscode_memleak.zip

There was no crash this time, perhaps because I did not have as many other memory-intensive programs running. Might be able to test this in a VM with a low commit limit later.

This log was created overnight with 3 large workspaces open in different windows. I'll try to narrow it down to a workspace I'm able to share.

[bpasero]

@Elusive138 it would be helpful to understand the details of the process that leaks. Can you find its PID and then print its full meta data using ps aux | grep <pid>?

[bpasero]

Ah, maybe this is on Windows, not sure :)?

[BobVul]

@bpasero The command line is:

"C:\Program Files (x86)\Microsoft VS Code\code.exe" --type=renderer --no-sandbox --lang=en-US --app-user-model-id=Microsoft.VisualStudioCode --node-integration=true --device-scale-factor=1 --enable-delegated-renderer --num-raster-threads=4 --gpu-rasterization-msaa-sample-count=8 --content-image-texture-target=3553 --video-image-texture-target=3553 --disable-accelerated-video-decode --channel="4896.1.1021371100\1043577992" /prefetch:673131151

Other details:

The process tree:

This is after another full day of use. As you can see, the process' memory usage appears to have risen linearly in that time (same workspace).

[BobVul]

Ah... this isn't even the really bad one. The second-level code.exe below it was worse.

4772 was the process that leaked overnight. The other one seems to have risen up due to my use during the day... I think.

[bpasero]

@Elusive138 this looks like you have multiple windows open with Code is that true?

[BobVul]

@bpasero Yes, the test last night had three windows open with different workspaces. I'll be trying it tonight with just one, on a clean Ext JS workspace as mentioned above.

[bpasero]

Sounds good!

[BobVul]

Unfortunately, no repro... this is odd. What could be causing a leak within that specific project?

Speaking of which, this might be different from @Tyriar's initial issue. His was unresponsive, mine and gwynjudd are crashes with the error dialog..? In which case maybe we should make a new issue...

[bpasero]

@Elusive138 odd. does it reproduce if you just open that workspace without opening any JS file?

Also, maybe we can start profiling this using the chrome developer tools where you can do heap snap shots. For that, just do a snapshot before and after some time to see where those memory goes.

[BobVul]

@bpasero Oh, I completely forgot about those devtools. I'll try that one tonight.

[BobVul]

@bpasero Heap snapshots of Main.js, workerMain.js and workerMain.js (2) don't really change. By maybe 5 MB at most.

[Tyriar]

Adding more on my situation, I tried to run it with a folder open (the one that crashed) but no working files and it didn't happen overnight. So it only happens when a folder is open and there are working files active.

[bpasero]

@Elusive138 we spawn other processes that can cause the memory pressure and they would not show up in the heap profile. However from one of your previous posts it seems you had identified the high memory coming from the renderer process and that one should show up under the heap snap shot. Are you sure you saw the high memory for the renderer process and not for its children? because the renderer is the parent of other processes that it spawns, e.g. a process for file watching.

@Tyriar can you be more specific what you mean by that. is it that you did not open any file in the editor or that you literally had the working files section empty?

The reason I am asking about opening a file or not is that for example the JS language service only starts once you open a JS file. The same is true for any other language service. So if this issue reproduces without opening a file in the editor it is likely that the memory leak is not in any language service but somewhere else.

[bpasero]

Another thing worthwhile to try: Run without any extension to see if maybe an extension is leaking. You can run with --disable-extensions to do so.

[BobVul]

@bpasero I'm sure the command line provided above was for 4772, the highlighted (leaky) one in the screenshot. It does say --type=renderer.

I'll leave it running over the weekend again and see what happens. I don't think I've done a clean test with no JS files open yet.

[Tyriar]

@bpasero I believe I had no file open (on the right), only working files above the tree. If I remember when I finish work I'll try check with a file open and note the language.

I don't know for sure but it probably wasn't a JavaScript file when mine crashed, more likely C++, Java or no language. I'll be sure to check.

[bpasero]

@Tyriar if it reproduces only by having something under working files and with all editors closed (right from the startup - that is important), we might be on to something.

[Tyriar]

@bpasero I might try open a bunch of instances under varying conditions before leaving work on Friday and see if I can get all of my experiments out of the way in one night. Unless you have reason to believe one instance of vscode hanging would break all the others?

[bpasero]

@Tyriar yes totally, if you have more than one window open it all adds up to the total memory being used and they all would crash. This is because when you open multiple windows, they still share all one parent process and one memory limit of something like 1-1.5 GB. To really understand where the leak is coming from it would be best to measure 1 window isolated under these conditions:

• no extension enabled via --disable-extensions (an extension that consumes lots of memory or leaks would also cause a crash)
• no file opened on startup (just closing a file after startup is already too late)
• no working file under working files

btw the only thing for working files that might have an impact is that for each working file that is not inside the workspace we install a file watcher to watch for changes. maybe this causes the leak, would be surprising though. also, we do not install a watcher if the file is inside the workspace that is opened, only for files outside.

[BobVul]

@bpasero No leaks when there are no open files (?). I'll try the clean workspace tonight after making sure to open a few.

[Tyriar]

I tried over the weekend with the following setup:

1. Launch with Code --disable-extensions
2. Open a ~1300 line JavaScript file

Memory and CPU were approximately the same Friday night and Monday morning.

[bpasero]

@Tyriar was this just opening an empty workspace with one file or opening a folder first and then a file from it?

[Tyriar]

@bpasero Opening empty workspace, I didn't open a folder and no folder was open when I launched

[bpasero]

Ok, good to know. So this seems to be related to having a (potentially large) folder open. Still to find out if just opening this folder is enough or having a file open only triggers it.

[BobVul]

@bpasero I had a large folder open with no files open over the weekend, with no noticeable increase in memory usage. I'll have the results of a large-folder-and-file-open test with hopefully a reproducible/shareable workspace in about 7 hours.

[Tyriar]

@bpasero the folder in which I experienced crashs before would have been 13000-14000 files strong, this includes the git repo which was around 2000 by itself.

I guess I'll double check that the crash still happens in the latest version next as it's been a while since I've seen it (due to testing).

[bpasero]

And I assume it is a JS workspace?

[Tyriar]

@bpasero py, cpp, js, html, css

[BobVul]

Ok, this seems to reproduce the slowly climbing memory usage, if on a smaller scale: VSCodeTest.zip. I had /ext/build/ext-all-debug.js opened.

(Yes, it's a 7z inside a zip... GitHub won't let me upload a 7z or xz directly, and zip and gzip are too big)

[bpasero]

@Elusive138 how much does it increase for you in avg? I had the workspace with some JS files open for 2 hours now without any increase in memory.

[BobVul]

@bpasero Sorry, I'm having trouble reproducing again. I think I had a stray git repo in an earlier attempt, that wasn't included in the archive above. I'll let you know when I have more concrete results.

Would be really helpful if it were possible to open multiple independent instances... being limited to one test a night is quite slow. Do you happen to carry across the Chromium flags for separate profiles?

[bpasero]

It is possible to build multiple Code versions that can run in parallel, though we have not documented it. If you change the values in https://github.com/Microsoft/vscode/blob/master/product.json so that they are different and then build from the command line, you can start multiple distinct instances. The identifiers that have to be unique are:

• darwinBundleIdentifier
• win32MutexName
• nameShort
• nameLong
• dataFolderName (e.g. "dataFolderName": ".oss-code-1")

[gushie]

Windows 8.1. In Code, opened a folder for a git repository containing 39,000 files in the morning. Did very little with it, a few small edits in a couple of files. By the end of the day the Commit Size reported by Task Manager was 672,164K. It grew steadily all day, even when I wasn't in the program.

[bpasero]

@gushie any chance this repo can be shared with me? also, what was the initial memory reported?

[gushie]

@bpasero I'm sorry, I'm not allowed to share the repo itself but if there are any details about it that might help, excluding the file contents themselves, let me know. The process initially starts at around 110MB, quickly grows to about 130MB before settling back to 90MB. It will then grow steadily from this point forward. (There are 5 other code.exe processes which vary from 4MB to 30MB but these don't noticably grow. There is only the one Code Window)

I don't know what the final Commit Size was from yesterday as the process had crashed when I returned to it today.

[BobVul]

@gushie I've been using Performance Monitor on the code.exe processes to monitor the "Private Size".

Pretty annoying that we can't share the workspaces we have issues with! I wonder if anyone's encountered this with something open-source. Looking for similarities: was your repo ever used with git-svn?

@bpasero Thanks, if the process I left hasn't repro'd when I get back in on Monday, I might try building a couple additional copies. Would need to figure out how to keep track of which one's which, though.. that might be a problem.