search

microsoft / nodejstools

Node.js Tools for Visual Studio
http://aka.ms/explorentvs
Apache License 2.0
1.8k stars 359 forks source link

System.OutOfMemoryException during analysis #678

Closed mousetraps closed 8 years ago

mousetraps commented 8 years ago

From @jwuliger

Splitting this off into a more targeted thread. 

    [Managed to Native Transition]  
    mscorlib.dll!System.IO.File.FillAttributeInfo(string path, ref Microsoft.Win32.Win32Native.WIN32_FILE_ATTRIBUTE_DATA data, bool tryagain, bool returnErrorOnNotFound)   Unknown
    mscorlib.dll!System.IO.Directory.InternalExists(string path, out int lastError) Unknown
    mscorlib.dll!System.IO.Directory.InternalExistsHelper(string path, bool checkHost)  Unknown
    Microsoft.NodejsTools.Npm.dll!Microsoft.NodejsTools.Npm.SPI.Package.IsMissing.get() Unknown
    Microsoft.NodejsTools.dll!Microsoft.NodejsTools.Project.AbstractNpmNode.ReloadHierarchy.AnonymousMethod__0(Microsoft.NodejsTools.Npm.IPackage module)   Unknown
    [Frames below may be incorrect and/or missing, no binaries loaded for System.Core.dll]  
    Microsoft.NodejsTools.dll!Microsoft.NodejsTools.Project.AbstractNpmNode.ReloadHierarchy(Microsoft.VisualStudioTools.Project.HierarchyNode parent, System.Collections.Generic.IEnumerable<Microsoft.NodejsTools.Npm.IPackage> modules)   Unknown
    Microsoft.NodejsTools.dll!Microsoft.NodejsTools.Project.AbstractNpmNode.ReloadHierarchy(Microsoft.VisualStudioTools.Project.HierarchyNode parent, System.Collections.Generic.IEnumerable<Microsoft.NodejsTools.Npm.IPackage> modules)   Unknown
    Microsoft.NodejsTools.dll!Microsoft.NodejsTools.Project.AbstractNpmNode.ReloadHierarchy(Microsoft.VisualStudioTools.Project.HierarchyNode parent, System.Collections.Generic.IEnumerable<Microsoft.NodejsTools.Npm.IPackage> modules)   Unknown
>   Microsoft.NodejsTools.dll!Microsoft.NodejsTools.Project.NodeModulesNode.ReloadHierarchy()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.InnerInvoke()  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.Execute()  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj)   Unknown
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot)    Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Unknown
    mscorlib.dll!System.Threading.Tasks.SynchronizationContextTaskScheduler.PostCallback(object obj)    Unknown
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)   Unknown
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Frames below may be incorrect and/or missing, no binaries loaded for WindowsBase.dll]  
    [Native to Managed Transition]

More info about environment at https://github.com/Microsoft/nodejstools/issues/663#issuecomment-175400231

jwuliger commented 8 years ago

Thanks for the help!

mousetraps commented 8 years ago

np - few questions that would be helpful in narrowing this down:

Thanks!

jwuliger commented 8 years ago

@mousetraps

Sure. So I am working on 4 different projects (all related to each other). Each one varies in its complexity and type. One of the projects I am not even using NPM modules.

It may be worth noting that I have overridden the default Visual Studio installations of node and npm to my global paths.

I am not doing anything special to the packages. I install them as they are.

It also for whatever reason does not have anything to do with NodeJS specific project types. Any project type you can think of that contains JavaScript is a problem. I am using AngularJS for these projects and even if I open a project as a "Website" it is still the same problem.

I do need Node/Express however to be my back-end. I am waiting to program those items as I had to uninstall NTVS. I also plan on hosting with Azure.

I know this was allot of info. I hope it helps. Let me know if you need anything!

Package Files.zip

mousetraps commented 8 years ago

@jwuliger Thanks - unfortunately I can't reproduce this issue at all. Any chance you could point me to some sample code or a project in which the issue occurs? Feel free to share privately - sitani at microsoft dot com

And just for clarification...

To isolate things a little further, then, could you try turning off Node.js IntelliSense and see if the issue still occurs?

f8k8 commented 8 years ago

I think I have a partial reproduction case for this. I tried debugging the NodeJS Tools with Visual Studio's Diagnostic Tools to monitor the memory usage, as I was getting a random crash in one of our projects every 10 minutes or so, even if Visual Studio was in the background not being used, and the event log points to an out of memory exception. I noticed that as the Analyser was running, every so often there'd be quite a large jump in memory usage of several hundred MB. It didn't take long before the memory usage was over 3GB, from a starting amount of around 500MB before the Analyser ran.

So to try and reproduce this minimally, I tried adding one of the larger modules to an empty project. So by creating a new project, and adding the 'lodash' module via npm, then just letting the Analyser run, you should be able to see the jumps in memory usage. They can take several minutes to happen, anywhere between 2-3 minutes and 10 minutes on my development machine, but when they do, the memory usage ramps quite quickly for a short time. They also seem to get progressively worse. I haven't had it crash yet with the test case, but after running for 30 minutes the process memory is up to 2GB after analysing just that one library. The runtime hasn't performed a Gen 2 GC yet that I can see though, so I'm not sure what the amount of that that can be reclaimed is.

I tried to debug the memory spikes, but without really knowing exactly how the Analyser works, I didn't get very far, but each time I broke into it during one of the jumps, it seemed to be to do with the call to AsStrongerUnion in DependentKeyValue.cs. The lines:

if (anyAdded && values.Count > projectState.Limits.DictValueTypes) {
     values = values.AsStrongerUnion();
}

Looking at values.Count and projectState.Limits.DictValueTypes, values.Count was usually in the 100+ range, and projectState.Limits.DictValueTypes is set to 30. The AsStrongerUnion call always seems to end up making copies of the AnalysisHashSet class, which contains several hundred entries, so my gut feeling is this is what's causing the memory issue, but as I said, I don't really know much about the internal workings of the Analyser, so I'm not 100% sure. It does look like a promising lead though.

The dependencies in my test are just:

"dependencies": {
    "lodash": "^4.5.1"
  }

and the main app.js file just contains the line:

console.log('Hello world');

mjbvz commented 8 years ago

I confirmed that the repo above is correct. Here is a memory chart of a project with lodash installed:

image

After five minutes, there's a huge spike for seemingly no reason where we literally triple memory usage. A second spike came later:

image

If you are running into this sort of memory problem, I recommend trying out the NTVS 1.2 alpha with the ES6 Intellisense Preview. It should be a better experience all around and ES6 Preview does not suffer the same performance problems.

AlexKeySmith commented 8 years ago

I too receive an out of memory exception, I see this in the event log.

Application: devenv.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at Microsoft.NodejsTools.Analysis.AnalysisSetDetails.AnalysisHashSet.EnsureSize(Int32)
   at Microsoft.NodejsTools.Analysis.AnalysisSetDetails.AnalysisHashSet.CheckGrow()
   at Microsoft.NodejsTools.Analysis.AnalysisSetDetails.AnalysisHashSet.AddOne(Microsoft.NodejsTools.Analysis.AnalysisProxy)
   at Microsoft.NodejsTools.Analysis.AnalysisSetDetails.AnalysisHashSet.Union(System.Collections.Generic.IEnumerable`1<Microsoft.NodejsTools.Analysis.AnalysisProxy>, Boolean ByRef)
   at Microsoft.NodejsTools.Analysis.AnalysisSet.AsUnion(Microsoft.NodejsTools.Analysis.IAnalysisSet, Microsoft.NodejsTools.Analysis.UnionComparer, Boolean ByRef)
   at Microsoft.NodejsTools.Analysis.AnalysisSet.AsUnion(Microsoft.NodejsTools.Analysis.IAnalysisSet, Int32, Boolean ByRef)
   at Microsoft.NodejsTools.Analysis.AnalysisSet.AsStrongerUnion(Microsoft.NodejsTools.Analysis.IAnalysisSet)
   at Microsoft.NodejsTools.Analysis.Analyzer.DependentKeyValue.AddTypes(Microsoft.NodejsTools.Analysis.ProjectEntry, Microsoft.NodejsTools.Analysis.JsAnalyzer, Microsoft.NodejsTools.Analysis.IAnalysisSet, Microsoft.NodejsTools.Analysis.IAnalysisSet, Boolean)
   at Microsoft.NodejsTools.Analysis.Values.ExpandoValue.SetIndex(Microsoft.NodejsTools.Parsing.Node, Microsoft.NodejsTools.Analysis.AnalysisUnit, Microsoft.NodejsTools.Analysis.IAnalysisSet, Microsoft.NodejsTools.Analysis.IAnalysisSet)
   at Microsoft.NodejsTools.Analysis.AnalysisSetExtensions.SetIndex(Microsoft.NodejsTools.Analysis.IAnalysisSet, Microsoft.NodejsTools.Parsing.Node, Microsoft.NodejsTools.Analysis.AnalysisUnit, Microsoft.NodejsTools.Analysis.IAnalysisSet, Microsoft.NodejsTools.Analysis.IAnalysisSet)
   at Microsoft.NodejsTools.Analysis.Analyzer.ExpressionEvaluator.AssignTo(Microsoft.NodejsTools.Parsing.Node, Microsoft.NodejsTools.Parsing.Expression, Microsoft.NodejsTools.Analysis.IAnalysisSet)
   at Microsoft.NodejsTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateBinary(Microsoft.NodejsTools.Analysis.Analyzer.ExpressionEvaluator, Microsoft.NodejsTools.Parsing.Node)
   at Microsoft.NodejsTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Microsoft.NodejsTools.Parsing.Node)
   at Microsoft.NodejsTools.Analysis.Analyzer.DDG.Walk(Microsoft.NodejsTools.Parsing.ExpressionStatement)
   at Microsoft.NodejsTools.Parsing.ExpressionStatement.Walk(Microsoft.NodejsTools.Parsing.AstVisitor)
   at Microsoft.NodejsTools.Analysis.Analyzer.DDG.Walk(Microsoft.NodejsTools.Parsing.Block)
   at Microsoft.NodejsTools.Parsing.Block.Walk(Microsoft.NodejsTools.Parsing.AstVisitor)
   at Microsoft.NodejsTools.Analysis.Analyzer.FunctionAnalysisUnit.AnalyzeWorker(Microsoft.NodejsTools.Analysis.Analyzer.DDG, System.Threading.CancellationToken)
   at Microsoft.NodejsTools.Analysis.Analyzer.DDG.Analyze(Microsoft.NodejsTools.Analysis.Deque`1<Microsoft.NodejsTools.Analysis.AnalysisUnit>, System.Threading.CancellationToken)
   at Microsoft.NodejsTools.Analysis.JsAnalyzer.AnalyzeQueuedEntries(System.Threading.CancellationToken)
   at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue+GroupAnalysis.Analyze(System.Threading.CancellationToken)
   at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue.Worker(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

It's a project I'm resurrecting at the moment and retrospectively created a visual studio project to use it's debugger.

Here is the package.json.

{
  "name": "XXX",
  "version": "1.0.0",
  "description": "XXX",
  "main": "gruntfile.js",
  "dependencies": {
    "grunt": "^1.0.1"
  },
  "devDependencies": {
    "grunt-bower-task": "^0.4.0",
    "grunt-concurrent": "^2.3.0",
    "grunt-contrib-jshint": "^1.0.0",
    "grunt-contrib-less": "^1.3.0",
    "grunt-contrib-uglify": "^1.0.1",
    "grunt-contrib-watch": "^1.0.0",
    "grunt-jsdoc": "^2.0.0",
    "grunt-karma": "^1.0.0",
    "grunt-ng-annotate": "^2.0.2",
    "grunt-notify": "^0.4.5",
    "grunt-open": "^0.2.3",
    "grunt-protractor-runner": "^3.2.0",
    "grunt-shell": "^1.0.0",
    "grunt-timer": "^0.6.0",
    "jshint-html-reporter": "^0.2.4",
    "karma": "^0.13.22",
    "karma-coverage": "^1.0.0",
    "karma-growl-reporter": "^1.0.0",
    "karma-htmlfile-reporter": "^0.2.2",
    "karma-jasmine": "^1.0.2",
    "karma-phantomjs-launcher": "^1.0.0",
    "load-grunt-tasks": "^3.5.0",
    "phantomjs-prebuilt": "^2.1.7",
    "protractor": "^3.3.0",
    "protractor-console": "2.0.1",
    "protractor-html-screenshot-reporter": "^0.0.21"
  },
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "XXX"
}

Here is the visual studio extensions installed:

Microsoft Visual Studio Enterprise 2015
Version 14.0.24720.00 Update 1
Microsoft .NET Framework
Version 4.6.01055

Installed Version: Enterprise

Architecture and Modeling Tools   00322-90000-61031-AA294
Microsoft Architecture and Modeling Tools

UML® and Unified Modeling Language™ are trademarks or registered trademarks of the Object Management Group, Inc. in the United States and other countries.

Visual Basic 2015   00322-90000-61031-AA294
Microsoft Visual Basic 2015

Visual C# 2015   00322-90000-61031-AA294
Microsoft Visual C# 2015

Visual C++ 2015   00322-90000-61031-AA294
Microsoft Visual C++ 2015

Visual F# 2015   00322-90000-61031-AA294
Microsoft Visual F# 2015

.NET Portability Analyzer   1.1.10808.0
API portability analyzer.

Add New File   3.2
The fastest and easiest way to add new files to any project - including files that start with a dot

ASP.NET and Web Tools 2015 (RC1 Update 1)   14.1.20203.0
ASP.NET and Web Tools 2015 (RC1 Update 1)

ASP.NET Web Frameworks and Tools 2012.2   4.1.41102.0
For additional information, visit http://go.microsoft.com/fwlink/?LinkID=309563

ASP.NET Web Frameworks and Tools 2013   5.2.40204.0
For additional information, visit http://www.asp.net/

Azure App Service Tools v2.7.1   14.0.11112.0
Azure App Service Tools v2.7.1

Common Azure Tools   1.6
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Devart Code Compare   4.1.78
Devart Code Compare
Copyright (c) 2012-2015 Devart. All rights reserved.
http://www.devart.com/codecompare/

Gardiner.VsShowMissing   1.0
Find files references in project files that don't exist on disk

GhostDoc   5.1.16036.0
GhostDoc automatically generates XML documentation comments.

GitHub.VisualStudio   1.0
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.

HideMenu   1.0
Hides the Visual Studio main menu, similar to Windows Explorer and Internet Explorer

Image Optimizer   3.5.84
Uses industry standard tools to optimize any JPEG, PNG and Gifs - including animated Gifs. Can do both lossy and lossless optimization.

JetBrains ReSharper Ultimate 2016.1.1    Build 105.0.20160504.101434
JetBrains ReSharper Ultimate package for Microsoft Visual Studio. For more information about ReSharper Ultimate, visit http://www.jetbrains.com/resharper. Copyright © 2016 JetBrains, Inc.

Microsoft Azure HDInsight HQL Service   2.0.2600.0
Language service for Hive query

Microsoft Azure HDInsight Tools for Visual Studio   2.0.2600.0
An integrated development environment for HDInsight application development.

Microsoft Azure Mobile Services Tools   1.4
Microsoft Azure Mobile Services Tools

Microsoft Azure Tools   2.7
Microsoft Azure Tools for Microsoft Visual Studio 2015 - v2.7.30818.1601

NCrunch   
Continuous Testing Tool for .NET
Copyright © 2010-2015 Remco Software Ltd

Node.js Tools   1.1.40329.04
Adds support for developing and debugging Node.js apps in Visual Studio

Node.js Tools - Profiling   1.1.40329.04
Profiling support for Node.js projects.

NuGet Package Manager   3.3.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

Open Command Line   2.0.168
Opens a command line at the root of the project. Support for all consoles such as CMD, PowerShell, Bash etc. Provides syntax highlighting, Intellisense and execution of .cmd and .bat files.

PreEmptive Analytics Visualizer   1.2
Microsoft Visual Studio extension to visualize aggregated summaries from the PreEmptive Analytics product.

ReAttach   1.1
Gives you an easy way to ReAttach to prior debugging targets. For more information, see http://visualstudiogallery.msdn.microsoft.com/8cccc206-b9de-42ef-8f5a-160ad0f017ae.

SQL Server Data Tools   14.0.50730.0
Microsoft SQL Server Data Tools

TypeScript   1.7.4.0
TypeScript for Microsoft Visual Studio

Visual Studio Spell Checker   VSSpellChecker
An editor extension that checks the spelling of comments, strings, and plain text as you type.
https://GitHub.com/EWSoftware/VSSpellChecker

Web Analyzer   1.7.75
Provides static analysis directly in Visual Studio for JavaScript, TypeScript, JSX, CSS and more

Web Compiler   1.10.306
Compiler for LESS, Sass and CoffeeScript files

Web Essentials 2015.1   1.0.211
Adds many useful features to Visual Studio for web developers. Requires Visual Studio 2015

Web Extension Pack   1.3.40
The easiest way to set up Visual Studio for the ultimate web development experience.

Xoreax IncrediBuild   7.1 (build 1659)
Xoreax IncrediBuild 7.1 (build 1659)
mjbvz commented 8 years ago

Thanks for the report. I recommend trying out the NTVS 1.2 alpha with the ES6 Intellisense Preview on VS2015 update 2. It includes a number of performance and memory leak fixes.

The Full Analyzer used in NTVS 1.1 has been a bug farm and accounts for nearly all of our crashes and most of the performance problems. That's why we are moving to using the Typescript analyzer with NTVS 1.2. This should be much more stable and will also pick up fixes and new features much more quickly.

I'm going to close this issue since we do have a solution in place with NTVS 1.2. Please give it a try and report any bugs or performance issues you run into. Thanks.