search

microsoft / nodejstools

Node.js Tools for Visual Studio
http://aka.ms/explorentvs
Apache License 2.0
1.8k stars 359 forks source link

Possible Memory Leak #264

Closed atom0s closed 8 years ago

atom0s commented 9 years ago

Some info on my system:

This is a fresh reformat as of 6 days ago. I have nothing on this system except for Visual Studio, MySQL (server and tools), and Chrome.

Here is the results of some testing:

Idle Launch On launch of Visual Studio, with no project loaded, I am idle at ~150MB of RAM usage by the devenv.exe

Open C# Application Solution Loading up a C# application solution, I peaked at ~300MB of RAM usage. Which after loading and a few minutes passed, some garbage collection occurred and dropped the usage down to about ~275MB.

Open Node.JS Solution Loading up a Node.JS solution immediately jumps the RAM usage to ~500MB while loading, and continues to climb almost consistently at ~100MB per second. At peak, I hit around 2.5GB of RAM usage with the amount constantly adjusting over the 2GB mark after the project has been fully loaded and the IDE is sitting idle. Image: http://i.imgur.com/lkfpqS9.png

Testing wise, I do believe the leak (if it even is a leak) is in the intellisense that the tools apply. If the Node.JS text editor settings are altered to completely disable the intellisense and remove the saving to disk, the usage idles out at about 400MB of RAM.

Not too sure if this is a leak or just how the intellisense works, but figured I'd report it since it was very noticeable on my laptop as it started to make the CPU fan spin louder while having VS open while using a NodeJS project.

kant2002 commented 9 years ago
  1. What version of NTVS you are using?
  2. How big you project code and how big is you dependencies?
  3. What Intellisense options is set: Full or Quick?
atom0s commented 9 years ago

I have Node.js Tools intellisense completely disabled now and do not have this issue anymore. However I am unsure what, if anything, I am losing by having it off since intellisense seems to be working well still.

mleanos commented 9 years ago

@atom0s, Intellisense is working for you within your javascript files as well, after disabling Intellisense? Or is it still working for you when working in HTML files?

atom0s commented 9 years ago

@mleanos - The intellisense that was taking over appears to have been from ReSharper. I forgot to disable it while doing the tests with the intellisense and such.

The intellisense that comes stock with NodeJS Tools is still the cause of the 2.5GB memory usage though for sure.

kant2002 commented 9 years ago

@atom0s Could you give some stats, like a) count of files, b) total lines in files without node_modules Would be good to know which dependencies your project have. Maybe this is easily reproduced just by adding dependencies

atom0s commented 9 years ago

A.) Roughly like 20-30 files, the project is a skeleton currently for building a website onto. Just some basic layout and getting express hooked up and ready. (8 total js files currently not including bower components and node_modules.)

B.) Javascript wise, maybe 500 lines of code so far. Not much is really done with the project yet.

As for dependencies: "body-parser": "~1.12.4", "cookie-parser": "~1.3.5", "debug": "~2.2.0", "del": "^1.2.0", "express": "~4.12.4", "jade": "~1.9.2", "less-middleware": "1.0.x", "less-plugin-clean-css": "^1.5.0", "morgan": "~1.5.3", "path": "^0.11.14", "serve-favicon": "~2.2.1", "vash": "^0.9.1"

And with our bower components we are using:

The application is an express 3 application.

atom0s commented 9 years ago

Another project I work on with this is much larger and is a backend API. Does not use express but the same memory usage occurs even though that project is much larger. (~30-40 dependencies, 50+ js files, 10k+ lines of code.)

mousetraps commented 9 years ago

44 includes some info that we'll need. In particular the diagnostic window and process explorer screenshot with cmd line arguments will be most useful right now.

kant2002 commented 9 years ago

I put information from memory dump in the #44 so everybody would be interested, could take a look. Here the reasoning and insights which I have from such simple investigation: Here the stripped data which I believe show important information

count       memsize type
....
  13114       209824 Microsoft.NodejsTools.Npm.SPI.NodeModules
  13137       210192 Microsoft.NodejsTools.Npm.SPI.BundledDependencies
....
  11659       233180 Microsoft.NodejsTools.Analysis.ModuleAnalysis
...
  11946       286704 Microsoft.NodejsTools.Parsing.JsAst
...
  11937       572976 Microsoft.NodejsTools.Analysis.AnalysisUnit
...
  52548       840768 Microsoft.NodejsTools.Npm.SPI.Dependencies
... 
  62990      2267640 Microsoft.NodejsTools.Analysis.ModuleTree
...
 194830      4675920 Microsoft.NodejsTools.Parsing.Block
...
 307867      6157340 Microsoft.NodejsTools.Parsing.ExpressionStatement
   91374      6213432 Microsoft.NodejsTools.Parsing.FunctionObject
 392565      6281040 Microsoft.NodejsTools.Parsing.ConstantWrapper
...
  375739      6980004 Microsoft.NodejsTools.Parsing.Expression[]
...
  399557     11187596 Microsoft.NodejsTools.Analysis.LocatedVariableDef
1049727   12596724 Microsoft.NodejsTools.Analysis.AnalysisProxy
  921073     14737168 Microsoft.NodejsTools.Analysis.VariableDef
...
  918157     18363140 Microsoft.NodejsTools.Parsing.Lookup
...
  467063     20550772 Newtonsoft.Json.Linq.JValue
...
  520358     33302912 Newtonsoft.Json.Linq.JProperty

and so one

From what I understand situation with analysis is following:

  1. We still analyze a lot of dependencies.
  2. We keep full Analysis Tree for all dependencies analyzed
  3. We keep not only Analysis Tree, but also parsing information which is not longer needed.
  4. JSON structure stored too.

Also question to the team: What is 

1601169     44832732 Microsoft.NodejsTools.Analysis.Analyzer.ReferenceableDependencyInfo

Why so much memory? and why storing JSON?

kant2002 commented 9 years ago

Once I realize that a lot of JSON stored I take a look on this and following uncover

... Skip other root
....
            ->  149f146c Microsoft.NodejsTools.Npm.SPI.GlobalPackages
            ->  149f2224 Microsoft.NodejsTools.Npm.SPI.NodeModules
            ->  149f2234 System.Collections.Generic.List`1[[Microsoft.NodejsTools.Npm.IPackage, Microsoft.NodejsTools.Npm]]
            ->  1b3c13b4 Microsoft.NodejsTools.Npm.IPackage[]
            ->  192fde88 Microsoft.NodejsTools.Npm.SPI.Package
            ->  19302480 Microsoft.NodejsTools.Npm.SPI.NodeModules
           ... skipped nested packages  ...
            ->  1931cb44 Microsoft.NodejsTools.Npm.IPackage[]
            ->  1930d530 Microsoft.NodejsTools.Npm.SPI.Package
            ->  1931052c Microsoft.NodejsTools.Npm.SPI.PackageJson
            ->  1930d564 Newtonsoft.Json.Linq.JObject
            ->  1930d5a8 Newtonsoft.Json.Linq.JPropertyKeyedCollection
            ->  1930d5bc System.Collections.Generic.List`1[[Newtonsoft.Json.Linq.JToken, Newtonsoft.Json]]
            ->  1930fddc Newtonsoft.Json.Linq.JToken[]
            ->  1930d5f4 Newtonsoft.Json.Linq.JProperty
... skipped nested Newtonsoft.Json.Linq.JProperty  ...
            ->  1930f1ec Newtonsoft.Json.Linq.JProperty
            ->  1930f22c Newtonsoft.Json.Linq.JProperty+JPropertyList
            ->  1930f238 Newtonsoft.Json.Linq.JObject

That's from simple project without any dependencies and just console.log() in the code. From that clearly seen that we convert all package.json files from NPMCache to our object model. On simplified model this is adds 40Mb. (thats from running !dumpheap -type Newtonsoft.Json -stat)

From the other 2 projects which I mention earlier in #44, JSON structure take 106Mb, so looks like we have one instance of GlobalPackages per each project. Which make sense, but not really required.

billti commented 9 years ago

We'll try and repro this and work on memory footprint for our Sept milestone.

brandon-arnold commented 9 years ago

Second this problem. I had the same issue with the VS 2013 Node tools a year ago and just came back to see the same behavior.

I created a public project for the developers to reproduce this behavior here, using a generic MEAN stack generator.

Please let me know if this helps or does not reproduce the behavior.

https://github.com/brandon-arnold/Vs2015NodeIssue

-Brandon

brandon-arnold commented 9 years ago

I see this is actually pretty much described by @mousetraps in #430 so you can ignore the above. Is there any way to force the Node project to ignore my bower components in client/bower_components/? The project should pick that up from the bower.json in my project root. And anyway, although node_modules/ is supposed to be ignored, it is still definitely causing memory/CPU usage to spike. If I delete node_modules/, things go back to normal.

mousetraps commented 9 years ago

@brandon-arnold

Indeed, you can add any directories you'd like to ignore to AnalysisIgnoredDirectories in your njsproj file. We actually ignore bower_components and core-js by default, though so I'm not sure that'll solve your issue: https://github.com/Microsoft/nodejstools/blob/master/Nodejs/Product/Nodejs/Microsoft.NodejsTools.targets#L12

This means anything containing that string will be ignored from analysis: https://github.com/Microsoft/nodejstools/blob/master/Nodejs/Product/Nodejs/Project/NodejsProjectNode.cs#L652-L656

node_modules is not supposed to be ignored (otherwise there's no way for us to provide intellisense!). It's simply excluded from the project due to project system performance issues (when you "include" something in a VS Project, that means it gets added to the msbuild file, and that's a lot of entries given how many files there are in node_modules or in bower_components.)

Also, somewhat unrelated, but just based on some of your comments I think you might find this page useful in providing more context on things :smiley: https://github.com/Microsoft/nodejstools/wiki/Working-with-Browser-(Clientside)-JavaScript-in-NTVS

brandon-arnold commented 9 years ago

Thanks @mousetraps. It makes sense that the NodeJS tools will still need to process the references for Intellisense. I will say, even after turning browser-side and Node-side Intellisense off, the CPU is still hit pretty hard. All I can guess is that the NodeJS tools do not scale well with a high number of included packages. The project I showed has 65 or so node modules. Are there some specific inefficiencies in how this stuff is being processed that you know you would fix, if you had the time to get to it? I would love to see these posted as Issues and take a look at it. We really need this.

mousetraps commented 9 years ago

@brandon-arnold Hmm... the CPU should not be hit very hard if there's no analysis going on... do you have any other extensions installed? e.g. ReSharper is known to be problematic with large projects.

OskarKlintrot commented 8 years ago

I too had this issue but after hitting the roof of max ram (when VS15 where using about 1,5-2,0GB, I only have 4GB in total) the ram drops down to about 700mb and then start to climb back up again. After disabling intellisens VS in now down to 280MB ram, which is normal on my PC. CPU is 0 % now when idle.

billti commented 8 years ago

I just installed the list of NPM packages outlined above in a new Express 4 app, and messed around requiring and coding against the packages, and I didn't see barely any CPU usage. Even after the NPM install, the analysis finished in the sub-second range, and memory seems quite reasonable too (below).

There must be something else going on here, as it doesn't seem to be endemic. Are you seeing this consistently, or specifically / mostly with the one project linked to above (I'll try with that next, but just wanted to ensure there wasn't something fundamentally broken with these packages first).

noleak

atom0s commented 8 years ago

Just tested it again, Node.JS tools set to use 'Full Intellisense' idle at around 300mb with nothing open. Shoots to over a gig once a nodejs project is loaded. Using the latest release package for Node.JS on VS2015.

atom0s commented 8 years ago

For some reason my phone didn't show your entire message so I replied without other info you requested.

When I set NodeJS to 'Full Intellisense' it happens every time I load my projects.

Here is the current package list of one of the projects that this still happens on: http://i.imgur.com/VtkQ4XF.png

ozziepeeps commented 8 years ago

I can confirm this same issue with Visual Studio 2015 Update 2, running Node.js Tools 1.1.40329.04. With full IntelliSense enabled my devenv.exe consumes memory until it dies. With reduced IntelliSense, I can get by.

johnnycamby commented 8 years ago

I would suggest to use 'Quick intelliScene' because that is what i 've been using for all my node.js projects and i always 've less memory consumption. But in the future i hope the VS-team l'l come up with a better solution to handle this issue.

NemoStein commented 8 years ago

Sadly, I'm stuck with 4gb of RAM in a Windows 10 (VS 2015) machine. Quick Intellisense is not enough in my case.

If I delete the .ntvs_analysis.dat file before opening VS project the leak takes longer to become a trouble, but when I open the project without deleting it, it is virtually impossible to work.

nthomson-choicescreening commented 8 years ago

I can confirm this issue and can possibly shed some light on what may be going on. I have a box that is running Windows 10, VS2015 and 4 GB of RAM. I have NodeJS 1.1. I have a project with about a dozen folders and running Jade. Well, my RAM will climb from a base of about 150 MB up to 2 GB when running the VS Node JS 1.1 Tools solution.

Here are some interesting finds:

So, there are workarounds but things seem to be pointing toward the loading of NPM packages into memory. I hope this gets fixed because I really like the NodJS tools solution for VS.

mjbvz commented 8 years ago

Please try upgrading to NTVS 1.2 Alpha and switching to ES6 Preview Intellisense. It requires VS 2015 update 2.

ES6 Preview Intellisense is generally much more stable and performant than our current full analyzer, and it offers a better experience pretty much all around. The full/quick analyzer is the root cause behind most of our crashes and most of the high resource consumption

Additionally, Alpha 2 of 1.2 fixed a number of memory leaks and should generally lower memory usage considerably (especially when dealing with projects with lots of NPM modules.) There's still room for improvement, so please report any issues you run into.

Thanks.

lookforit commented 8 years ago

@mjbvz Yes,after switching to ES6,it turns to be ok.

michaeldunnOMM commented 8 years ago

Moving to NTVS 2 has done absolutely nothing. After opening a node project, I see my memory usage rocket up to 2,500,000 K before it grinds to a halt and crashes.

mjbvz commented 8 years ago

@michaeldunnOMM Sorry to hear that. We made some improvements but definitely want to identify and fix any remaining issues in this area. I just want to confirm that you are on ES6 Preview IntelliSense when you see this.

Another hunch is that you may be running into a MAX_PATH issue. We have some mitigation steps to address those problems here: https://github.com/Microsoft/nodejs-guidelines/blob/master/windows-environment.md#max_path-explanation-and-workarounds Let me know if you have any questions about them.

It would probably be best if you could open a new issue so we can better track this new investigation. Please let us know:

The above points are all available in: Help -> About Visual Studio and there's even a handy Copy Info button.

Also:

Thanks.

michaeldunnOMM commented 8 years ago

Thanks for the reply! ES6 Intellisense Preview is currently disabled as an option. I cannot work out why. I have followed your instructions as per the MAX_PATH issue which haven't helped I'm afraid. Here are the details you requested.

Microsoft Visual Studio Community 2015 Version 14.0.23107.0 D14REL Microsoft .NET Framework Version 4.6.00081

Installed Version: Community

Visual Basic 2015 00322-20000-00000-AA664 Microsoft Visual Basic 2015

Visual C# 2015 00322-20000-00000-AA664 Microsoft Visual C# 2015

Visual C++ 2015 00322-20000-00000-AA664 Microsoft Visual C++ 2015

Application Insights Tools for Visual Studio Package 1.0 Application Insights Tools for Visual Studio

ASP.NET and Web Tools 14.0.20626.0 ASP.NET and Web Tools

ASP.NET Web Frameworks and Tools 2013 5.2.30624.0 For additional information, visit http://www.asp.net/

Common Azure Tools 1.5 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

GenerateUnitTest 1.0 Generates unit test code for methods in classes under test.

GruntLauncher 1.1 Right click extension to launch Grunt commands

Indent Guides 14 Indent Guides

Adds visual guides at each indentation level.

Microsoft Azure Mobile Services Tools 1.4 Microsoft Azure Mobile Services Tools

Node.js Tools 1.2.40329.01 Adds support for developing and debugging Node.js apps in Visual Studio

Node.js Tools - Profiling 1.2.40329.01 Profiling support for Node.js projects.

NuGet Package Manager 3.0.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

PreEmptive Analytics Visualizer 1.2 Microsoft Visual Studio extension to visualize aggregated summaries from the PreEmptive Analytics product.

SQL Server Data Tools 14.0.50616.0 Microsoft SQL Server Data Tools

Web Essentials 2015.2 2.0.214 Adds many useful features to Visual Studio for web developers. Requires Visual Studio 2015

mjbvz commented 8 years ago

@michaeldunnOMM Thank you for the info. The MAX_PATH issue is probably a false lead, I just threw it out as a possibility since it is another common cause of memory and performance problems. The memory issue you are running into is almost certainly because you are on the static old analysis engine.

The ES6 IntelliSense option is probably grayed out because you do not have TypeScript 1.6 plus installed. The NTVS 1.2 Alpha releases also require Visual Studio 2015 Update 2 or later to function properly. We don't show the proper warning to alert uses to this problem in the alpha (#1003), but have tried to fix this for the next release.

I recommend that you upgrade to VS 2015 Update 2 and give this another shot. If you still see the ES6 IntelliSense option grayed out, check that TypeScript is listed as a Visual Studio installed package.

Hope that helps. Please let me know if you have any questions or run into more problems.