Crash when running overnight

[Tyriar]

Ubuntu 12.04, VSCode 0.10.1

Several times VS Code has become unresponsive overnight on the above configuration (locked). Here is the program output:

$ code .
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

<--- Last few GCs --->

173527197 ms: Scavenge 1397.0 (1457.6) -> 1397.0 (1457.6) MB, 1.8 / 0 ms [allocation failure] [incremental marking delaying mark-sweep].
173527199 ms: Scavenge 1397.0 (1457.6) -> 1397.0 (1457.6) MB, 1.9 / 0 ms [allocation failure] [incremental marking delaying mark-sweep].
173529040 ms: Mark-sweep 1397.0 (1457.6) -> 1396.9 (1457.6) MB, 1841.7 / 98 ms [last resort gc].
173530775 ms: Mark-sweep 1396.9 (1457.6) -> 1396.1 (1457.6) MB, 1735.0 / 5 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x8a48933a859 <String[7]: file://>
    1: _completed [file:////home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/vs/workbench/workbench.main.js:~1544] [pc=0x23ff9b465433] (this=0x1a37262790b1 <JS Object>,e=0x1cd36e9041b9 <undefined>)
    2: arguments adaptor frame: 0->1
    6: bound  [native v8natives.js:1208] [pc=0x23ff99a26270] (this=0x8a489346089 <JS Global Object>)

==== Details =============================...

Failed to get crash dump id.
Report Id: 
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: channel closed
    at process.target.send (internal/child_process.js:509:16)
    at Console.console.error (/home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/bootstrap.js:5:937)
    at process.<anonymous> (/home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/bootstrap.js:5:1340)
    at emitOne (events.js:77:13)
    at process.emit (events.js:169:7)
    at process._fatalException (node.js:223:26)
[VS Code]: detected unresponsive

This has never occurred with Atom.

[gushie]

@Elusive138 We use Atlassian tools, so SourceTree connecting to a private Stash server

[BobVul]

Ah, that's completely different from mine. Was a local Git repo (w/ Git Extensions) connecting to an SVN server. My current test is with a git repo based off the vscode one... hopefully that'll repro and I can share it.

[bpasero]

@gushie does it reproduce if you just open the workspace without opening any file? try to first close all files and restart to get this setup. if that does not reproduce, does it reproduce if you open the workspace and open one JS file and let it like that?

[gushie]

@bpasero I had already left it open with just a single unedited working file last night, and checking this morning the memory usage does't appear to have changed much. I've now edited and done a build on the file, and will leave this to see what happens.

Note until I read through this issue, I hadn't ever cleared down the working files. (I hadn't really had a need to, I generally ignored this panel and switched files via the project file explorer underneath it, or with Ctrl+E).

[Tyriar]

Crashed again, this time it was after regular usage but it was left in this state:

• ~14000 file folder open
• 5 working files active (.cc, .java, .h, .py, .json)
• No file open in text editor

ps aux output:

❯ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
daniimms  6086  0.0  0.5 970620 92180 pts/0    Sl   Jan11   0:36 /home/local/ANT/daniimms/VSCode-linux-x64/Code /home/local/ANT/daniimms/VSCode-linux-x64/resources/app/out/bootstrap --type=watcherService

As mentioned though this was after regular usage and not in safe mode so it could have been due to an extension?

[gushie]

I left a single file with a small edit open, just that file in the working files. Otherwise untouched all day. Memory has not increased from it's initial value.

[bpasero]

@Tyriar well the process you are showing is our file watching service and on Linux/Mac we use Chokidar for it (https://github.com/paulmillr/chokidar). This would indicate that they have a leak when watching files. Is there some build task constantly running and changing files?

@gushie was this without opening a folder?

[gushie]

@bpasero I did have the folder open. There is obviously some action that appears to kick off the memory leaks beyond the initial opening of a file within the folder. I'll continue to monitor and try and narrow it down.

[Tyriar]

@bpasero That was the only vscode-related process I could see in the output, it's possible that it wasn't visible in the output because it was already killed due to unresponsiveness? As shown that process was only using 0.5% of memory and 0% of CPU. I think out of my recent discoveries the issue I'm experiencing probably isn't due to a JS service.

[bpasero]

Btw even if we do not find the reason for this until end of month, I pushed a change so that the crash dialog you get by default selects an action to reopen the window so that you can continue to work where you left. This is also good for the case where you have multiple windows open and only one of them crashes.

[bpasero]

(one thing missing and for the future is to be able to periodically flush UI state to disk so that reopening also restores your previous work environment)

[Tyriar]

@bpasero :+1: that would be helpful.

[nojvek]

VS has been crashing every morning for about a month now. I have come to the end of my patience.

I hope this is being treated as a PRI 0 bug because any sane editor should not crash this often.

Please let me know how I can get a trace out about why this is happening. I have seen this happen on other machines as well.

[bpasero]

@nojvek are you able to share with me the workspace where this happens?

[nojvek]

I'm not sure what you mean by share workspace. It's a typescript/HTML/CSS project with a bit of c++. Around ~1500 files and 80,000 lines of code.

Is there a way in VS to show crashdumps?

[bpasero]

@nojvek I would like to have a workspace that I can run Code with to reproduce this issue. We suspect it is related to a memory leak somewhere, it is just not clear where. If you can send me a zip of the workspace or maybe the git URL if it is open source.

[nojvek]

Benjamin,

Its internal Microsoft source code and I don't think I can give it to you.

If there is some memory monitoring tool I can add let me know.

Would it work if i take memory snapshots with the chrome debugger embedded in vscode every couple of hours to see what is going on?

On Saturday, January 23, 2016, Benjamin Pasero notifications@github.com wrote:

@nojvek https://github.com/nojvek I would like to have a workspace that I can run Code with to reproduce this issue. We suspect it is related to a memory leak somewhere, it is just not clear where. If you can send me a zip of the workspace or maybe the git URL if it is open source.

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-174192367.

[bpasero]

Yes memory snapshots will help if and only if this leak is inside the main side of VS Code. However, it seems that at least in one case the leak was in one of the processes we spawn (the file watcher). Maybe you could identify using process explorer, which process is constantly increasing in terms of memory consumption.

If it helps to provide me with the source code: I work for Microsoft ;)

[nojvek]

I'll have a word with my manager. This is in Windows repo so I'll have to be a bit careful.

From the thread it seems the chokidar is used for the file watcher. A while back I played with the source code. It was a bit of spaghetti.

I'll see if I can load chokidar on its own simple node app and see if it's the culprit.

Btw any specific reasons why vs needs to use chokidar? There are other cross platform file watchers right?

On Sunday, January 24, 2016, Benjamin Pasero notifications@github.com wrote:

Yes memory snapshots will help if and only if this leak is inside the main side of VS Code. However, it seems that at least in one case the leak was in one of the processes we spawn (the file watcher). Maybe you could identify using process explorer, which process is constantly increasing in terms of memory consumption.

If it helps to provide me with the source code: I work for Microsoft ;)

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-174269028.

[bpasero]

Chokidar is used on Mac and Linux, if you are on Windows, we use a C# file watcher. I am happy to accept a PR with a working and performant cross file watcher solution :)

[nojvek]

You know as of node 4.0 the fs.watch is stabilized on both windows and mac. See: https://github.com/Microsoft/TypeScript/issues/4643.

Typescript has a pretty solid cross platform file/directory watching logic. https://github.com/Microsoft/TypeScript/blob/931d162620c7e09377c12875834e1838c4cdd51b/src/compiler/sys.ts

I'll try to get a local build of VS code with a change and see if it still crashes. If it doesn't then I'll definitely send a PR.

[bpasero]

I know it got much better by using the recursive watch call on the directory and thus avoiding having to attach a watch listener per folder, but I think the events still sometimes miss the file/folder name, at least in some cases. I feel more confi using C# here.

[nojvek]

I managed to get a repro today. I just crashed before I was going to pull up Developer tools console to take a memory snapshot.

http://i.imgur.com/rirFul1.png

It seems the typescript server is taking about 200MB. The renderer consumed about 800MB and then crashed.

When I get about 500MB of usage, I'll attempt to take a memory snapshot. The --renderer process is the webkit process right?

This was after two days since I last saw a crash. It seems using the tool for a 8 hour period editing lots of TS files adds up to usage faster.

[bpasero]

Nice one, the renderer process is the main process for the editor. If you can do a memory snapshot there it would bring us further. Unfortunately the snapshot would not work for any other of the processes, but those seem to be ok'ish.

[nojvek]

Is it normal for tscse (typescript server) to consume 200MB though?

On Tue, Jan 26, 2016 at 9:29 PM, Benjamin Pasero notifications@github.com wrote:

Nice one, the renderer process is the main process for the editor. If you can do a memory snapshot there it would bring us further. Unfortunately the snapshot would not work for any other of the processes, but those seem to be ok'ish.

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-175408302.

[bpasero]

Yeah, very easily.

[nojvek]

Lol! I was in the process of taking heap snapshot and VS crashed.

I will try again and see if I have better luck tomorrow.

Is there a way to tell vscode to increase its memory limit so it doesn't crash while trying to take a heap snapshot ?

[bpasero]

I think what happens is that taking the snapshot you caused an out of memory exception :-/. There is no way to change the memory limit, it is hardcoded in V8.

[BobVul]

Maybe try to take the snapshot a bit earlier, before memory usage grows so high? Is it a gradual increase?

I've not been able to reproduce (no crashes, reasonable memory usage) for the last two weeks or so. Only difference I can think of is the PowerShell extension updating from 0.1.0 to 0.3.1 -- but I don't have any PS files in the workspace anyway, and it crashed with extensions disabled at the time.

[nojvek]

I got a successful dump of around 300MB used by the --renderer process. It crashed 10 seconds after I took the dump.

It seems the bulk of the usage is in the DOM tree.

https://www.dropbox.com/s/dk5exyke3fqgahk/VS.heapsnapshot?dl=0

Hope this helps.

[bpasero]

Hmmmm, I wonder if the profiler is telling the real truth here. I see all the tree elements and we might as well have a leak in the tree, but I would be surprised that this leak could cause an out of memory in such a short period of time.

[nojvek]

I took a few more dumps.

It seems opening .min.js files causes a huge jump in memory. Even after closing the files the memory doesn't seem to go down. Between all the memory dumps it seems DOM is the dominator.

Also not sure why there were two renderer processes. May be one of them was the chrome devtools. But I got very close to 1 gig.

https://www.dropbox.com/sh/3t238zx7l3vfpzx/AABCbFBNYLzHinkN7OTe6ubya?dl=0

I took a screenshot of the two processes consuming ~500MB of ram.

[bpasero]

Yes, one is devtools. It is expected that memory goes up when you open files. But I would like to understand how a VS Code that is running over night without activity increases in memory use until it crashes.

[nojvek]

Well the repro is: Start vs code. Open one or two js/ts files on a large folder with lots of .tsconfig files. Do some scrolling / editing. Close all working files. Put it in the even at 200c overnight. Cake!

On Friday, January 29, 2016, Benjamin Pasero notifications@github.com wrote:

Yes, one is devtools. It is expected that memory goes up when you open files. But I would like to understand how a VS Code that is running over night without activity increases in memory use until it crashes.

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-177087988.

[BobVul]

@nojvek Are the .tsconfigs necessary to repro? 'cause I've had crashes in the past without anything TS-related at all. Though I can't even repro those anymore...

[nojvek]

Our project just happens to have them. Could be a red herring.

On Saturday, January 30, 2016, Bob notifications@github.com wrote:

@nojvek https://github.com/nojvek Are the .tsconfigs necessary to repro? 'cause I've had crashes in the past without anything TS-related at all. Though I can't even repro those anymore...

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-177198518.

[gushie]

The only .* files we have in our folder are the .git folder and .gitignore

[bpasero]

@nojvek are you using a custom typescript server with the project, or the out of the box TS we ship in 0.10.6? The one we ship in 0.10.6 is TypeScript 1.7.5

[tinganho]

@bpasero @nojvek if it is a memory issue on the JS side. You can diff the heap snap shot on Chrome.

1. Take a snapshot.
2. Do the action that causes memory leak.
3. Force garbage collection. Otherwise the next heap snapshot will include the objects.
4. Take another snapshot.
5. Diff both snapshot.

Everything can be done with in Chrome Devtools above. The force of garbage collection is in the timeline tab, the garbage icon. And the diffing is done by just choosing it on the snapshot explorer on the filter menu.

Just report which objects is retained and possible retain paths(just below the heap table). And that is all the data needed to fix this issue.

Don't need to wait a whole day to collect data about a crash :wink:

[nojvek]

The Dropbox link I sent had three heat snapshots taken half an hour apart. I didn't do the force garbage collection. Will give that a trick.

On Sunday, January 31, 2016, Tingan Ho notifications@github.com wrote:

@bpasero https://github.com/bpasero @nojvek https://github.com/nojvek if it is a memory issue on the JS side. You can diff the heap snap shot on Chrome.

1. Take a snapshot.
2. Do the action that causes memory leak.
3. Force garbage collection. Otherwise the next heap snapshot will include the objects.
4. Take another snapshot.
5. Diff both snapshot.

Everything can be done with in Chrome Devtools above. The force of garbage collection is in the timeline tab, the garbage icon. And the diffing is done by just choosing it on the snapshot explorer on the filter menu.

Just report which objects is retained and possible retain paths(just below the heap table). And that is all the data needed to fix this issue.

Don't need to wait a whole day to collect data about a crash [image: :wink:]

— Reply to this email directly or view it on GitHub https://github.com/Microsoft/vscode/issues/508#issuecomment-177462412.

[milenkovic]

I have same problem, VS Code is constantly crashing, not only over night, even after 30min of being idle. I am using Windows 7, 64bit and latest version of VS Code. Regarding project itself it's huge js project, with node_modules and bower_components also loaded. Totally it has around 40K of files.

Steps to reproduce: 1) Open project folder(with lot of files), open several files so they could be in Working Files list. 2) Open task manager. 3) Put cursor to focus some file in VS Code 4) Watch as memory rises during time.

[bpasero]

Can people with JS workspaces seeing this issue give our 0.10.7 insiders release a try (https://vscode-builds.azurewebsites.net/insider) with the option to use Salsa as JS language service: https://github.com/Microsoft/vscode-docs/blob/vnext/release-notes/latest.md#javascript---salsa-preview

Thanks!

[lordgreg]

+1 for this issue. Also appearing on Windows. Its totally annoying and I don't like it.

[gwynjudd]

@bpasero I would like to try out the new build, but whenever I try to access the link you provided (https://vscode-builds.azurewebsites.net/insider), I get a very unhelpful error message page with the following text:

Sign In Sorry, but we’re having trouble signing you in. We received a bad request.

Additional technical information: Correlation ID: e477dc1b-c193-4cf9-864b-1d3fdbba4f34 Timestamp: 2016-02-03 19:37:17Z AADSTS50020: User account 'gjudd@example.com' from identity provider 'https://sts.windows.net/5a7d8144-6966-4b1e-8147-de672a307ea0/' does not exist in tenant 'Microsoft' and cannot access the application '9d5f02f6-ffd9-4e80-92d5-e42c85e09bc9' in that tenant. The account needs to be added as an external user in the tenant first. Sign out and sign in again with a different Azure Active Directory user account.

I'm not able to download the build and try it out because of this issue.

[bpasero]

Ah sorry, wrong link, please use https://code.visualstudio.com/insiders

[gwynjudd]

@bpasero thanks I have the insiders build running with Salsa as the JS language service. I'll let you know how it goes over the next day or so

[milenkovic]

@bpasero, I tried insiders release and problem still exists, after one night in idle VS Code has crashed.

[bpasero]

@milenkovic @gwynjudd you will also have to follow the steps to enable Salsa as outlined in https://github.com/Microsoft/vscode-docs/blob/vnext/release-notes/latest.md#javascript---salsa-preview

[milenkovic]

@bpasero, You were right, enabling Salsa fixed the problem. I left VS Code opened trough weekend and it still didn't crashed.

[bpasero]

Good to hear that! Would be interesting if others could verify this too by trying.