search

dotnet / project-system

The .NET Project System for Visual Studio
MIT License
969 stars 387 forks source link

Enormous memory and CPU consumption by SDK project style #4078

Closed evgeny-burmakov closed 4 years ago

evgeny-burmakov commented 6 years ago

After migrating from classic project structure to SDK style on large solution we are met with enormous memory consumption by VisualStudio and performance degradation (mainly on starting phase but quite long time). Also VS crashes with OutOfMemoryException time to time when project files are actively modified. Memory consumption increased from 1.2Gb on classic style to 2.3Gb on SDK style.

Environment:

panopticoncentral commented 4 years ago

The first step would be to look at the memory dump. It looks like it's still uploading to the system, so once we've got it we can take a first look and see what's what. That would probably suggest the next step.

drewnoakes commented 4 years ago

@DocteurRe thanks for filing the feedback ticket. I've hit a problem with tooling while trying to open the dump (https://github.com/microsoft/perfview/issues/1220). I'll get back to you as soon as possible.

panopticoncentral commented 4 years ago

Looking at the dump, it's clear that you are being impacted by https://github.com/microsoft/msbuild/issues/3668 (FYI, @ladipro) which we are prioritizing fixing ASAP.

How many projects are in the solution? This could help in interpreting the MSBuild objects on the heap. And a question I ask everyone with large solutions: are you generating your solution, or did you build it by hand?

lifengl commented 4 years ago

Thanks for the dump. We have 398 instances of CPS projects, 752 instances of CPS configured projects, 756 instances of evaluated projects, and 1245 copies of project instances in the dump. So: 1, We have a clear code path to leak project instances (created a PR) 2, it looks to me that all configurations of projects are loaded into memory, without earlier configuration switching. Typically, that means an incomplete solution leading the solution to load & remapping solution/project configurations. Very often, it indicates the solution files are generated by a tool? Is that the case? Do you get an updated solution file, if you happen to have a chance to save it after loading the solution? The solution asks the project system whether each configuration is deployable, and unfortunately, the new project system has an extensibility model to ask MEF extensions to answer this question, which leads it to load everything, which is different from the old one. We discussed potential solutions (of course, it would break existing extension model), but not yet done the work. Usually, fixing the generation tool would eliminate that issue.

drewnoakes commented 4 years ago

@DocteurRe what version of the SDK are you using? We made a perf change in 16.7p3 (SDK 3.1.4xx) to significantly reduce the number of MSBuild items reported during design-time builds. These items are kept in memory. If you're using an older SDK, you'll still have all these items in memory. Unfortunately I still cannot access your dump (have asked Lifeng and Paul to see how they've managed to do that) so cannot tell whether you do or do not have all these items in memory myself.

DocteurRe commented 4 years ago

@drewnoakes I created another ticket using VS16.7.0 preview 3.1 : https://developercommunity.visualstudio.com/content/problem/1095043/enormous-memory-and-cpu-consumption-resulting-a-cr-1.html

lifengl commented 4 years ago

@drewnoakes : i am using the 64bit perfview from @sharwell, and can open the dump without running into problems. Can you tell me how to verify whether the dependency tree memory improvement is activated in the dump file? Also, I can conform the issue that the issue @panopticoncentral identified earlier (globbing info) was a top issue in the heap memory usage, and we should get it fixed.

@DocteurRe : can you help us to understand why excessive number of configurations are loaded into memory. A mismatched solution file can be the reason, or it might be caused by an extension. Some behavior difference between the old and new project system may lead some extensions to do excessive things. I noticed from the dump that GhostDoc is handing reference changes. The new project system doesn't resolve reference during the initial loading time, when DT builds finishes & NuGet restoring is done, the references collection will be refreshed, and may lead excessive work in some of those extensions (so they may need wait for operation progress and start to process data until the project state is stable?)

jaytonic commented 4 years ago

@lifengl Since I work on the same project than @DocteurRe , maybe I could answer this:

We have ~400 projects, each project has several target(Any cpu, x86, x64, multiplied by release+debug option). x86 and x64 is a left over of the years of work on this project, this could be deleted(so we could end only with 2 configuration). This SLN isn't generated. But it has been merged by 20 different people over the past 10 years, so I cannot exclude 100% there is something wrong in it.

How many configuration do you see?

lifengl commented 4 years ago

I think loading excessive configuration is the top thing we need address, because loading two configurations for a project almost double the memory foot-print of project memory usage (not include other part of VS). It gives lots of benefit for the fix.

The solution/project mapping section looks like the block here (https://github.com/dotnet/project-system/blob/master/ProjectSystem.sln). 

GlobalSection(SolutionConfigurationPlatforms) = preSolution
    Debug|Any CPU = Debug|Any CPU
    Release|Any CPU = Release|Any CPU
    Description = The .NET project system for Visual Studio.
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
    {9FBD7EF2-9449-486D-9FDD-FA56160AA8BB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
    {9FBD7EF2-9449-486D-9FDD-FA56160AA8BB}.Debug|Any CPU.Build.0 = Debug|Any CPU
    {9FBD7EF2-9449-486D-9FDD-FA56160AA8BB}.Release|Any CPU.ActiveCfg = Release|Any CPU
    {9FBD7EF2-9449-486D-9FDD-FA56160AA8BB}.Release|Any CPU.Build.0 = Release|Any CPU

Bascially, it list {each_Project_Guid}.[each solution configuration name].[a few values] = [Project configuration name]

If one solution configuration is not mapped for a specific project, or the project configuration does not exist in the project, it will have to recreate the list. Usually, once the solution is loaded, and saved. It will fix itself. But if your solution keeps running out of memory, it might be broken, and maybe check it manually might help.

If it doesn't, it is also possible that an extension/feature you are using might load configurations when they accesses configuration related properties. To find that out, maybe taking some PerfView ETW trace would help to capture that. (The loading time is long, so it is a little bit trick to know that, but we can give a try.)

drewnoakes commented 4 years ago

Joining the dots: This analysis helped identify #6327 which is being addressed in 16.8.

@jgrossrieder @DocteurRe were you able to look into Lifeng's questions around configurations?

lifengl commented 4 years ago

@jgrossrieder @DocteurRe: some of the issues identified either through the investigation related this thread, or investigations have been made in 16.8 update preview 1, which has been released recently. Those fixes, however, may not fix problems for your specific solution, or they might not make enough improvements for your scenario. To make sure we are on the right track to make progress in the upcoming update (or a late one), can you help to try the new preview update with your solution? If you still run into out of memory, can you send a new feedback ticket similar to the last time? It will help us to drive a new round of investigation.

Thanks

DocteurRe commented 4 years ago

@lifengl I created another ticket using VS16.8.0 preview 1.0:

https://developercommunity.visualstudio.com/content/problem/1147028/enormous-memory-and-cpu-consumption-resulting-a-cr-2.html

Comparing to VS 16.7.0 preview 3.1, the behavior is a lot better on this VS16.8.0 preview 1.0: I opened Visual Studio, then started recording, then opened our SLN solution file and wait for full loading. Then I have done a “Rebuild solution”. Memory increased up to about 5300 MB, but Visual Studio didn’t crash, and I was able to work.

The current behavior is not perfect as the memory usage is still huge, and everything is a bit slower than when using old style .csproj, but now Visual Studio is usable.

Great job guys 😃 Thanks a lot for all your efforts

lifengl commented 4 years ago

@DocteurRe : thanks a lot to take the time to try the product, and sent another memory snapshot. I verified that the excessive number of project instances problem was gone in the new dump. We will take further look to find out other area to improve the memory consumption problem.

davkean commented 4 years ago

I'm going to close this out. We have a internal working group tracking and reducing memory footprint of Visual Studio, especially around solution load, branch switches, and solution close.

For folks running into issues in the future, please Help -> Send Feedback -> Report a Problem recording a trace of 20 seconds and we'll take a look.

ladipro commented 4 years ago

The ticket was assigned to me as the memory consumption was suspected to be build system related. I have analyzed the dump and closed the ticket with the following observations:

+1 on what @davkean wrote: We are still actively working on improving this.

jaytonic commented 4 years ago

@ladipro Thank you for improving this, because, since VS Dev team has choosen to publish a x64 version, we are quite limited in space. Resharper still brings us a lot of help, and we will not deactivate it.

Where is the rest (760MB) of the managed heap?

At the current state, we are still not able to work with VS Studio and projects in the new formats(even it has been highly improved). It's even more a pain because we have a lot of RAM on our computers, but VS just cannot use it, and our projects will not decrease their size in the future.

ladipro commented 4 years ago

The rest of the heap is taken up by other VS components - the project system, Roslyn/compilers, etc. I don't think I can speak authoritatively about their footprint as the numbers can easily be misinterpreted. For example, looking at Microsoft.VisualStudio.ProjectSystem. objects I see that exclusive retained bytes is 270 MB. But since the project system holds evaluated projects, this includes the 190 MB of MSBuild objects. The object graph is complex.

What I can point out with certainty is that 110 MB of the managed heap is wasted by duplicated strings and we have an ongoing effort to eliminate these.

@jgrossrieder the dump in the last ticket was large but wasn't close to the 4 GB process limit. Would it be possible to get us an OOM or close-to-OOM dump from the latest VS preview?

panopticoncentral commented 4 years ago

One trick that is helpful is to use ProcDump to capture a dump at the point at which the process runs out of memory. I tend to do something like procdump -w -ma -e devenv which waits for devenv.exe to start, monitors it for fatal exceptions, and then captures a full memory dump when a fatal exception occurs.