Please Send Halp 💥

[Mike-E-angelo]

Describe the bug: Actual real-world usage of Razor-based solution runs into scaling concerns and makes one doubt long-term scalability. I've sent out a plea here: https://developercommunity.visualstudio.com/t/Razor-Tooling:-Ginormous-Memory-Usage-Af/10146575#T-N10247503

But I am opening an issue here for greater visibility.

Version used: Any within the past two years. They all feature the same grief.

To reproduce:

1. Make a solution with ~150k lines of C# and over 35k lines of Razor:
2. Open solution (this is an older solution that does not have as much code but should still easily present the issues/challenges for you).
3. Do development
4. Observe grief

Expected behavior: Several GB of RAM used, normal and effective GC to keep around such levels.

Actual behavior: Nearly 8-12 GB ultimately used with frequent GC and subsequent grief.

Additional context: As mentioned in the plea/link above I am running (or attempting to run 😁) a Microsoft for Startups-accepted startup based on your technology and am now actively generating revenue. However, this has gotten to the point where I am beginning to doubt the long-term viability due to this and many other issues I am facing in Visual Studio. The runtime tech works great but the tooling is massively challenging and has been for several years now.

I am heavily invested in Microsoft/Blazor/Razor so I can put up with this friction for a while longer, but I cannot see myself realistically convincing any other developer to endure the challenges that the environment poses, especially if they are from a React/Node/etc background. I am certain they would tell me I am insane for putting up with this situation -- which, incidentally enough, is exactly how I feel ATM -- and would recommend switching to another tech stack altogether.

FWIW I have reached out to @NTaylorMullen and the suggestion was to reach out to @davidwengier (👋) which I suppose I could have done privately but I feel more comfortable doing this here with this issue. 😁

I am wanting to explore ways to improve my solution so that it better fits your current design so that I do not encounter constant existential battles of doubt from using your technology. The obvious stab is to reduce the number of projects used. I did try this at one point but it did not make any difference. 😭 I could have been doing something non-obvious, however, so this is why I am reaching out now to make sure I have all considerations factored.

Thank you for any consideration and/or assistance you can provide. 🙏

[davidwengier]

You probably don't want to hear this, but "we're working on it" :)

Unfortunately there isn't one thing we can do to fix the issues, but you should know that perf (and reliability) is our top focus, across tooling and compiler teams. I suspect most of the improvements will come from the compiler side of things, and the good news there is that we have some execellent plans (removing design-time/run-time file split, re-using source generator assets, removing the usage of the "dynamic file" system) and even better news is that the compiler is now part of this repo, and owned by the same team that owns the VB/C# compiler (and they seem to know what they're doing). Unfortunately all of this is going to take a long time, and whilst there will definitely be incremental improvements (eg https://github.com/dotnet/razor/pull/8056, https://github.com/dotnet/razor/pull/6515, https://github.com/dotnet/razor/pull/8067, etc.), its impossible to say exactly when your issues in particular will be "fixed".

To share some recent results though, @chsienki has recently been doing on the source generator to improve perf, that has some very exciting results:	Method	Job	BuildConfiguration	Mean	Error	StdDev	Median	P95	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
Razor_Add_Independent	Baseline	Release_Nuget	183.429 ms	3.0090 ms	7.8207 ms	181.514 ms	205.752 ms	1.00	0.00	14000.0000	1000.0000	-	62,397 KB
Razor_Add_Independent	Current	Release	4.847 ms	0.0671 ms	0.0594 ms	4.827 ms	4.955 ms	0.03	0.00	312.5000	281.2500	-	1,957 KB
Razor_Edit_Independent	Baseline	Release_Nuget	14.477 ms	0.1900 ms	0.1777 ms	14.412 ms	14.794 ms	1.00	0.00	843.7500	406.2500	-	5,087 KB
Razor_Edit_Independent	Current	Release	4.457 ms	0.0641 ms	0.0568 ms	4.448 ms	4.554 ms	0.31	0.00	328.1250	296.8750	-	1,980 KB
Razor_Remove_Independent	Baseline	Release_Nuget	176.081 ms	3.2093 ms	8.3981 ms	172.740 ms	201.633 ms	1.000	0.00	13000.0000	1000.0000	-	61,412 KB
Razor_Remove_Independent	Current	Release	1.245 ms	0.0152 ms	0.0119 ms	1.244 ms	1.263 ms	0.007	0.00	156.2500	82.0313	-	814 KB
Razor_Edit_DependentIgnorable	Baseline	Release_Nuget	2.747 ms	0.0519 ms	0.0460 ms	2.755 ms	2.796 ms	1.00	0.00	226.5625	156.2500	-	1,146 KB
Razor_Edit_DependentIgnorable	Current	Release	3.129 ms	0.0586 ms	0.1128 ms	3.095 ms	3.349 ms	1.15	0.03	210.9375	187.5000	-	1,292 KB
Razor_Edit_Dependent	Baseline	Release_Nuget	137.331 ms	4.3744 ms	12.8980 ms	130.231 ms	166.848 ms	1.00	0.00	9000.0000	500.0000	-	41,554 KB
Razor_Edit_Dependent	Current	Release	5.938 ms	0.0996 ms	0.0932 ms	5.904 ms	6.091 ms	0.04	0.00	343.7500	312.5000	-	2,148 KB
Razor_Remove_Dependent	Baseline	Release_Nuget	187.709 ms	8.4931 ms	24.9087 ms	172.975 ms	232.938 ms	1.000	0.00	14000.0000	1000.0000	-	61,193 KB
Razor_Remove_Dependent	Current	Release	1.219 ms	0.0144 ms	0.0120 ms	1.213 ms	1.242 ms	0.005	0.00	156.2500	82.0313	-	814 KB

At the moment, these are still being worked on, so aren't in the repo, and unfortunately only help with Hot Reload scenarios, but as I mentioned above, we have plans and are working towards reusing the source generator assets so these improvements will come to tooling in time. I just don't know what time.

I just wanted to mention all of this so that you know we are working on it, and mostly so that you know we really appreciate the effort you've gone to in sharing your data, and reporting problems, and the patience that you have shown thus far.

[Mike-E-angelo]

You probably don't want to hear this, but "we're working on it" :)

You'd be surprised @davidwengier I very much do want to hear that. 😁 Thank you for the update and insight with your provided metrics. It would of course be ideal to have a general window of time when this will be completed but it's understandable that these are not available at the moment. Hoping+wishing for the best here and that you are able to provide some quality relief soon. 👍

[davidwengier]

Working on some more perf improvements in https://github.com/dotnet/razor/pull/8133, in particular one change has allocations in document mapping down from something that linearly increases with file size, to just a few kb regardless of file size. Some of those improvements only affect light bulbs, but the document mapping in particular will help a lot of things, including even just clicking around in an un-changed document.

[Mike-E-angelo]

Awesome, thank you @davidwengier any bit helps, so any updates like this are very valuable to me.

FWIW please also consider that in my world it's a war on two fronts. In addition to Visual Studio, I am also running R# which has also been struggling in this area as well for several years now. Tracking my woes on JetBrains' side here: https://youtrack.jetbrains.com/issue/RSRP-484052/2021.1-RTM-Razor-Development-R-Doubles-CPU-Utilization

One may ask why I am such a glutton for punishment. Believe me when I say I ask myself this constantly. 😅😭 To be honest, if the runtime didn't work so freaking well I would have serious thoughts here about my direction. However, the finished product when deployed works very well to my liking, so that really only leaves the tooling as the biggest risk & consideration in my world.

The deeper issue for me is that I have had my nose in .NET since 2001, and this is by far and away the most successful endeavor I have set out on -- even if it's only a few hundred dollars ATM -- so I would like to protect the 20-plus-year investment, so to speak. :)

[Mike-E-angelo]

Hi, @davidwengier thank you again for looking into this issue. A thought arose for me over the weekend and I wanted to ensure that I brought it up here. Can you confirm if the code that I have provided is being used anywhere to validate the performance improvements being made on your side?

The fundamental concern here is that you are addressing one issue while my codebase is facing another.

Primarily, what concerns me is that opening my solution takes many, many minutes (5-10) of CPU churn by devenv.exe and ServiceHub.RoslynCodeAnalysisService.exe and will ultimately take many GBs of memory right out of the gate, around 7-8GB.

Additionally, when I close the solution in Visual Studio and force a GC, all the Razor objects are retained in memory as denoted in this issue. This has been the case for Razor development since the outset so want to confirm if this is being addressed in your current efforts.

I may be misunderstanding the issue, of course, but my understanding is that such a large amount of memory utilization -- which only increases as development goes on -- is leading to a good amount of GC activity and a lot of the grief that I am experiencing whenever Razor development is encountered.

Thank you for any further update/insight that you can provide. 👍

[davidwengier]

The fundamental concern here is that you are addressing one issue while my codebase is facing another.

I completely understand the concern, and you are correct to think this. The problem is that your codebase is unlikely to be facing a single issue, and this is especially true when dealing with memory allocation issues as they are (obviously) cummulative. Because of that, there isn't going to be one fix that will magically solve all of the issues, but rather the solutions will be cumulative too. So for example, I made a change that will definitely stop memory bring being consumed on every keystroke by the light bulb feature of VS in Razor files. The investigation of that issue, and the validation of the fix, was done in isolation for the light bulb feature. I cannot say if that will stop memory being consumed in VS as a whole, for your solution, on your machine. I would love it if I could, but its simply impossible to know. All we can do is keep fixing things, and making things better.

[Mike-E-angelo]

Thank you for your reply and thoughtful consideration @davidwengier. I appreciate the approach of taking things as they arrive. However, from my perspective, I have been doing this for over two years now and it does not seem to be getting any better. This is part of my challenge here.

It would seem that at the very least your team would have a codebase that is similar to mine (i.e., 100k+ lines of C#, 25k+ lines of Razor) that would surface these issues, which are very clear, present, and obvious. In my opinion, these issues should not be making it into Preview/RTM bands but have been doing so for well over a year now.

Right now as it stands, I am basically a full-time QA engineer for your product(s) and this does feel very fair to me as someone trying to build a business on your technology.

Consider that as a result of this, I am battling two fronts:

1. The inordinate challenge of actually making money with a technology
2. The unnecessary challenge of tooling used to make the technology that makes the money in the first challenge

So you are aware, the experience is so bad for me that it is starting to impact my estimates now when I plan new features. Development that should take about 1 day is being given 1.5 days now to account for all the friction that tooling incurs at the moment. This is neither desired nor long-term viable and leads me toward the desire to start reaching for another technology stack altogether. Again, I am so invested and therefore committed -- one might say attached 😁 -- at the moment that this is not a serious desire. However, the longer this continues to persist and feels like a sinister IQ test for how much grief one will endure, the more realistic this desire becomes.

Thank you for any further consideration and discussion around this.

[Mike-E-angelo]

Again, airing out some grief here, but I will have mornings this one where I spend about 10 minutes developing with a new preview, and notice that my CPU is perpetually pegged at 30-35%. I then need to spend over an hour tracking it down and getting it reported, making sure it's not something that I have already reported as there are already dozens of these it seems I have reported over the past several years:

https://developercommunity.visualstudio.com/t/Perpetual-35-CPU-Utilization/10267474

Again it would seem if someone over there simply took a solution with a good amount of code (100k lines of C#, 25k+ lines of Razor) and did some basic development for 30 minutes or so, they would easily find issues like this with a release candidate before it gets shipped it out to a band.

One final thought here is that when I start looking for investors. A lot of startups do not use Microsoft tech as you know and I am sure to get questions about this. I am having trouble seeing if they would actually approve of what I have to go through to make everything work here. This is a concern emerging in my world.

This is also in addition to finding another developer that would actually be able to tolerate what I have to go through at the moment to get the solution to build+debug, and upon successfully building+debugging, having to deal with issues such as the above. Even if I was able to manage to get someone to accept this, I am not sure they would last for long and retention+turnover would be a very real challenge. 😭

[adrianwright109]

@Mike-E-angelo have you tried disabling all your third party VS plugins/extensions (if you have any) like ReSharper etc as I wonder if they might be causing the high CPU usage?

[Mike-E-angelo]

This is the first thing I do before reporting @adrianwright109 :) As I mentioned I have several years of experience now fighting the two fronts and have gotten good at knowing where to report the grief.

(Additionally, I also attach a dotTrace screenshot of the hotpath in the issue description, and it is not in JetBrains/R#-related assemblies)

[Mike-E-angelo]

As I mentioned I have several years of experience now fighting the two fronts and have gotten good at knowing where to report the grief.

Speaking of which, just encountered a new issue in ServiceHub.RoslynCodeAnalysisService.exe:

https://developercommunity.visualstudio.com/t/Perpetual-30-CPU-Utilization-in-Servic/10267489

Yes, seeing double this morning. 30-35% utilization in devenv.exe, followed by another in ServiceHub.RoslynCodeAnalysisService.exe 😭

[davidwengier]

Thanks for the comments folks, I appreciate the passion and definitely appretiate the frustration at things not working. Having said that, I'm not entirely sure that there is anything I can really say in response. Yes @Mike-E-angelo we have various projects of various sizes and complexities we use, and you've been kind enough to provide yours, and all of the tracing and diagnostics that you record are great. The issue here is not about finding issues - we agree that they exist - the issue is that the fixes are long and complicated. If someone, or something, has given you the impression that anyone is denying you're having issues, or that there are performance improvements that we need to make, then let me be very clear and categorically state that is not true.

I wish we could snap our fingers and fix everything in the next version, but the realities and complexities of the fixes cannot be understated. For example, the thing that causes the most memory issues (at the moment) is in the Razor compiler, with various things like how it uses syntax annotations, conditional weak tables, and how it mixes syntax and semantics etc. The fix here is to overhaul the compiler to bring it in line with the latest architecture from Roslyn, but not only is that a lot of work, it also would break compatibility with every public API the current compiler has, and break 1st and 3rd party consumers. Rolling out these sorts of breaking changes necessarily has to be an effort over a reasonable time, coordinated with lots of partner teams.

Hopefully each month with each preview release there will be more and more fixes and improvements, but we're definitely on a "slowly chip away at things" tragectory. The bad news is we're not finished, the good news is we're not pretending we are.

[Mike-E-angelo]

Thank you very much for your continued dialogue @davidwengier. I promise that at some point I'll get out of your hair here. :)

If someone, or something, has given you the impression that anyone is denying you're having issues, or that there are performance improvements that we need to make

It's not that I feel ignored. Quite the opposite actually and it's quite nice on that front. :) My primary beef is that I will report one issue and another one will pop in its place. It feels like I have been faced with the same bug wearing different masks now for two years straight. As I said, it feels like an IQ test of how much grief someone will endure and I am passing it spectacularly -- or failing, depending on your perspective. 🙃

Anyways, I appreciate you being here and letting us know we're heard and you are chipping away at things. Again, if the final deployed result didn't work so very well in production this would be a no-brainer for me, and I would have switched to another stack long ago. Having excellent performance in production/runtime is really what's keeping me holding onto hope here. 👍

FWIW, in other news, JetBrains fixed another big nasty in my world, so that will help a bit here with the chips as you chip away over there. 🙏

[Mike-E-angelo]

Not sure if this one is already captured somewhere, but noticed nearly a minute of CPU churn after pressing a few keys in my editor this morning. This is 2x the usual fare. Reported here:

https://developercommunity.visualstudio.com/t/Razor-Development:-Several-Keypresses-Ta/10276767?port=1026&fsid=7a71e5fb-e57b-4e75-8bfc-f432324b281a

Also, worth noting that the GitHub issue here is being tracked in my semi-weekly standups in the Friction Points section where I discuss items causing me distress: https://youtu.be/jSQxW0DJjtI?t=851

I hope I pronounced your name correctly, @davidwengier 😬 I may just call you "Mr. David" from now on. :)

[davidwengier]

Close enough :)

In other news, another perf improvement just went in, which has some very good results for improving formatting time and memory. Even if you never explicitly format a document, you'll still get some benefit from this PR whenever you paste, as at the moment the editor is always issuing a formatting request when pasting (tracked by #7025)

[Mike-E-angelo]

Another baddie gets tagged: https://github.com/dotnet/razor/issues/8371#issuecomment-1451528446

[Mike-E-angelo]

Thank you for your continued dialog @davidwengier it is appreciated.

I am enduring some grief here this morning, and I wanted to be sure about the ticket I referenced in the opening message of this ticket. I realize there are several issues that are being tracked. However, the one that is generally causing me the most harm is high RAM usage that does not appear to be properly swept. This ultimately leads to high RAM utilization and constantly disruptive GC sweeps.

The IDE does not start out like this, so to me, this seems to imply a memory leak somewhere.

My inquiry with this here is that this seems like a bug that can be addressed sooner rather than later, as opposed to a fundamental issue with tooling that might take a while. I could be misunderstanding this, of course, so wanted to pipe up here and get clarification if possible.

For additional reference for my morning here, I am facing GC sweeps roughly every 20 seconds ATM with basic typing in my environment, which has 32GB of RAM dedicated to it via Hyper-V Manager. 😬

[Mike-E-angelo]

FWIW JetBrains is looking into some of the issues I am encountering and they are confirming a 1GB memory leak in Visual Studio when opening/closing the v3.zip solution as provided/reported to you:

https://youtrack.jetbrains.com/issue/RSRP-491473/R-Razor-Development-EAP-UI-Freezes#focus=Comments-27-6927940.0-0

[davidwengier]

I wanted to be sure about the ticket I referenced in the opening message of this ticket

The main culprit in your screenshots seem to be TagHelperDescriptors, which is a bit of a problematic area. Firstly the good news is that the issue is being actively worked on (eg, https://github.com/dotnet/razor/pull/8369) by the best minds in the business, but there is slightly more to it as far as a long lasting solution goes.

The TL;DR is that tag helper descriptors are something of a communication mechanism between tooling and compiler. The compiler "discovers" them by finding all components in the project, referenced projects, and references assemblies, and then sends that list over to the IDE tooling. The IDE tooling caches them, so it can hand the list back to the compiler when it needs to compile a Razor page. The tooling also looks after invalidating that cache (when files/projects change) and asking the compiler for a new list (or rather, a delta) if necessary. At runtime some amount of work has to be re-done because runtime compilation of Razor is slightly different, and driven by a source generator.

All of this communication means lots of object allocations, serialization, deserialization etc. Long term what we're aiming for is to converge on a single compilation of Razor for design time and runtime, and have the source generator running for both scenarios. This would mean the entire compilation model of the IDE is flipped on its head, and instead of the IDE telling the compiler when it needs to do work, the source generator ("the compiler") would be run by Roslyn and it would provide data to the IDE. This should remove a bunch of the communication requirements for tag helpers (in theory, everything except the bare minimum needed for IntelliSense), but is a rather large undertaking with many steps. Rest assured we are actively working on those steps, on both the compiler and tooling teams.

[Mike-E-angelo]

Thank you for the update and for your comments @davidwengier, it is greatly appreciated. After hitting those GC sweeps, I took a look at my Hyper-V configuration and noticed that while I had 32GB allocated to my development instance, it was a max allocation using dynamic memory. The instance had 16GB allocated to it from a starting configuration.

I did some housekeeping and reconfigured the utilized RAM from my other instances, and now my development instance has 32GB starting allocation with dynamic going to 40GB. I developed yesterday and my environment seemed a bit nicer without the constant+obvious sweeping that I was encountering several days prior. I emphasize seemed as that whenever I mention any success around these issues is when they usually get worse. 😩

For the time being, it looks like I have bought myself some further breathing room and can attend to the next fire here.

Thank you for letting me know about #8369. It is beyond awesome for me to see the effort. 👍💖🙏😌

[Mike-E-angelo]

So I have been using Visual Studio 17.6.0 Preview 2.0 since its availability last Tuesday and it has been quite remarkable. I have noticed no GC sweeps and RAM utilization is usually around ~10GB. Additionally, forcing GC after closing all documents will get RAM utilization to ~5GB which seems new. Usually, once it gets to 10GB (or higher) it will stick there and forcing GC will not matter.

It's gotten to the point that I wanted to stick my neck out here and see if #8369 is the reason for this magic. 🤔

[Mike-E-angelo]

@davidwengier / @DustinCampbell / @ryzngard / @sharwell / @333fred / @jaredpar

I wanted to take some time to ensure things are as they seem before commenting and closing out this issue. Everything is really amazing now, and it feels like a new IDE for me. I still get 25% CPU utilization for 30-60 seconds at a time with a single keypress, but that pales in comparison of having GC sweeps occur every 30 seconds and freezing my IDE for 30 seconds at a time.

My IDE now reports 5GB of usage via R# memory reporting consistently. Sometimes even dipping in the high 4GB range. It will jump up to 10-15GB sometimes but it's brief and drops down to 4-5GB after a while with typical GC. There are no longer frequent+disruptive UI freezes and everything feels/seems like it should be. 😌🙏

I can easily develop in this environment now, and I no longer think of having made the wrong life choice in choosing my current development path. Everything seems possible again and I am elated with my current orientation. 🙏🙏🙏

What really cements this for me is the great communication and effort here by the team there. I am beyond grateful that you take the time to listen to your customers and fix the important parts when necessary. It took a really long time to address this after multiple attempts with many various previous issues, but in the end you finally fixed the root issue ailing me during all this time, and that means everything to me. You have me for life. I mean, you had me already, but this super cements me in, as I said.

Thank you again. Please know that you and your efforts are making a difference out there. 👍💖🙏

[davidwengier]

Thank you for the kind words @Mike-E-angelo, we're glad things have improved, but I thnk I can speak for everyone when I say: I hope you don't mind if we keep working to improve things 😁