Roslyn Produces Abnormally High CPU Utilization When Many Files Are Open

[Mike-E-angelo]

Version Used: This is seen as of Visual Studio 2022 Preview 7

Steps to Reproduce:

1. Open this solution
2. Expand the DragonSpark.Application project so that the DragonSpark.Application/Security/Identity/Profile namespace is visible.
3. Open all the visible files under the project with Right-click -> Open (this is about 60 of them):
4. Open this file, pressing Enter and Backspace several times at the end of this line
5. Observe CPU usage and time (it should be nominal and expected)
6. Now repeat step 3 <-- Triggering Event
7. Repeat step 4
8. Repeat step 5 -- CPU utilization should be abnormally utilized in both amount and time

Expected Behavior: The expected behavior is seen with steps 1-5. It is expected that steps 6-8 produce the same behavior as well, but they do not.

Actual Behavior: While steps 1-5 take several seconds to complete at ~6% CPU utilization (nominal and expected), steps 6-8 take up to nearly 30 seconds to complete, with double the amount of CPU utilization during this time.

Additional notes This is (thankfully) a very easy reproduction to produce the issue. Until I was able to reproduce it using this method, I was only encountering it in impossibly complex situations where massive refactoring occurred and many files were opened. This is still the case today and I actively encounter this problem when significant refactoring is involved. When paired with issues such this one it becomes quite the challenge to keep Visual Studio open.

Note that this is also captured in developercommunity here with provided ETL/DMPs. An issue was never made here in the Roslyn repository to properly track so doing that now.

Please do let me know if you have any further questions and I will do my best to assist you. Thank you for any consideration towards addressing this issue, and for all your excellent work over there. 🙏

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ldsenow]

I have the same issue as well. The CPU is always at 50%+ even when idle.

[CyrusNajmabadi]

Based on those traces. It is likely InheritanceMargin or CodeLens. If you have those two running, can you disable and see if it clears things up for you?

[ldsenow]

It seems to work if I disable CodeLens. I will monitor this and see if this is false positive

[CyrusNajmabadi]

This doesn't excuse it. But that would at least match the trace, and would match how CodeLens is currently designed. It effectively runs int eh background in response to changes, looking for references to all the items. Computing references is not cheap (and we've discovered that SourceGenerators can highly exacerbate this), as it effectively involves searching potentially all downstream candidates. So as long as it is sitting there constantly asking to be brought up to date after an edit, you'll constantly see high CPU.

We do not have any super good ideas (yet) on how to reconcile the fundamental tension between Find-Refs being expensive, and Codelens constantly wanting to show up-to-date Find-Refs info directly in the UI.

[Mike-E-angelo]

Thank you for taking the time to investigate this issue @CyrusNajmabadi and @ldsenow.

Unfortunately, what @ldsenow is experiencing appears to be a different issue altogether. Or at least, in this case, I am still able to easily reproduce this issue by disabling both CodeLens and InheritanceMargin and following the reproduction steps listed above.

As described, Roslyn operates nominally and as expected in steps 1-5 with many files opened. It isn't until step 6 is encountered that a state issue of some sort is triggered, and abnormal CPU utilization/time is then consistently observed.

Additionally, I am able to run into this issue when any sort of wide refactoring occurs (additional example may be found here) but reaching this state is more complicated/involved than the simple reproduction steps outlined above.

[CyrusNajmabadi]

I'll look at your traces again today @Mike-E-angelo however, they an showed work being done in find refs. So something is calling that. The likely subjects are code lens and inheritance margin. But it's possible it's some other feature or extension (as find refs is a public api we expose)

[CyrusNajmabadi]

Tagging @NTaylorMullen lots of hits in Razor in the user reports linked here: https://developercommunity.visualstudio.com/t/Significant-CPU-Utilization-Involving-Bo/1566277#T-N1570838

[Mike-E-angelo]

Thank you for that, @CyrusNajmabadi. Full disclosure here: developercommunity.com/DragonSpark is me, so I am responsible for those reports as well. FWIW, I attempted to change my username to align more closely with my GitHub name for better discernment over three weeks ago, but ran into an apparent bug with developercommunity that is preventing me from changing it.

Yes, there are a lot of fires in my world. 😁

Anyways, to ensure a better understanding of this issue, there are two scenarios that do invoke it:

1. It can (easily -- thankfully) be reproduced without any Razor components by simply opening many (50-60+) .cs files twice in Visual Studio. (tracked with this ticket)
2. It is also observed when massive refactoring occurs, and Rrazor components do appear to agitate this scenario. This is tracked with this ticket

[CyrusNajmabadi]

Following the above (OP) instructions on:

produces no strange behavior for me.

[CyrusNajmabadi]

It can (easily -- thankfully) be reproduced without any Razor components by simply opening many (50-60+) .cs files twice in Visual Studio. (tracked with this ticket)

Can you open another perf trace and submit it and link me to it? I'm not seeing any problem locally, and the problem i identified before doesn't seem to be the one you're hitting.

Note: please have inheritance margin and codelens off so we can see what's consuming the CPU in that scenario.

[Mike-E-angelo]

Great, thank you very much for investigating this @CyrusNajmabadi.

I did some more digging on my side. It appears there is a gotcha with CodeLens. That is, once this scenario is triggered, it does not matter if you turn CodeLens on or off. When I tested earlier, the triggering event (step 6) had already taken place in my IDE instance, so my IDE was already "stuck" regardless if I turned CodeLens on or off.

However, after starting a new IDE instance with CodeLens turned off, this issue does not present itself. If I turn CodeLens back on and re-trigger it and turn off CodeLens, the issue still persists.

So, this does appear to be a CodeLens integration issue, and once it's triggered, it appears that it is turned "on" until you restart Visual Studio.

I am curious, are you able to reproduce this on your side with CodeLens turned on?

Again you should see that Visual Studio operates fine until you re-open the set of files for the second time. Be aware that this is rare, but in the past, I have had this not occur on the 2nd attempt. This has not been the case for me recently and I have been able to consistently get this to present on the 2nd opening of files. However, if you are not seeing it occur on the 2nd attempt on your side, you may need to try a 3rd or 4th. The telltale sign is that after re-opening the set of files, ServiceHub.RoslynCodeAnalyzerService.exe will spend a lot more CPU+time when re-opening the files than it does on the first attempt.

Now that we know it's a CodeLens issue, it still bothers me that it works fine the first time after opening the files, but doesn't after it's triggered via the 2+ times of opening (and persists whether CodeLens is enabled or not). I would like to hear your thoughts on this. Also, I am assuming that since this is now squarely identified as a CodeLens integration issue, that another capture is not required. Please do let me know how you would like to proceed.

[CyrusNajmabadi]

Thanks for the additional info. I'll try repro'ing this next week with codelens on.

[Mike-E-angelo]

I wanted to check in on this issue. Please let me know if there is any further information I can provide to help further diagnose, and/or if another capture is required on my side to get additional information, and I will do my best to assist.

To summarize, this appears to be related to CodeLens, but once it occurs and CodeLens is then disabled, the issue still persists.

[sharwell]

but once it occurs and CodeLens is then disabled, the issue still persists

From past observations, I suspect CodeLens doesn't completely turn off until you restart the IDE.

[Mike-E-angelo]

Bummer @sharwell but thank you for that tidbit. I am interested in workarounds for when I encounter this problem, and restarting the IDE seems to be it which is about as disruptive as the issue itself. :)

Is this the correct repository for CodeLens or is there another location I should be reporting?

[CyrusNajmabadi]

This is the correct repository for the parts of codelens related to roslyn (like 'reference count' which is the part you're hitting). So this issue should stay here. Other parts of codelens (like the source control information) would live outside of here.

[AqlaSolutions]

I have the same issue in VS 17.1.0 Preview 5.0

[Mike-E-angelo]

Yeah, my major IDE fires have quelled down a bit so this one is no longer as much of a priority. It would still be neat to get it addressed, of course, as it is very reproducible with the provided steps. 😇 Outside of that, this issue was more like a cymbal crash when things were going very Very Badly™ during development. Nowadays, I have issues like https://github.com/dotnet/razor-tooling/issues/5697 which still occur but things seem to be progressing into "intermittently poor" from "always poor" development environment/experience. :)

[sec]

I can confirm also having high CPU (and RAM) usage after upgrade to 17.1.0 (by ServiceHub.RoslynCodeAnalysisService.exe)

[CyrusNajmabadi]

@sec please follow the instructions here: https://github.com/dotnet/roslyn/blob/main/docs/wiki/Reporting-Visual-Studio-crashes-and-performance-issues.md#performance-issues so file a trace so we can diagnose which component is causing the problem. Thanks!

[Mike-E-angelo]

Thank you for all the replies everyone. I am glad to see that I am not the only one experiencing this. I can confirm that this still occurs in a consistent reproducible fashion on my end using the steps as described in the original post. I have attached a recording of this in the associated developercommunity ticket here using 17.2 Preview 2 for your review here:

https://developercommunity.visualstudio.com/t/servicehubroslyncodeanalysisserviceexe-high-cpu-ut/1482376#T-ND1695016

[CyrusNajmabadi]

@Mike-E-angelo Do you have code-lens still no? I'm seeing this all in the find-refs codepath. Which indicates that either this, or some similar feature, is on and is continually recomputing references after you make edits.

If you turn code-lens off and restart do you still see the issue?

[CyrusNajmabadi]

Note: if this is codelens, then this is likely by-design. You're in a config where you have things setup to show the results results of a very expensive computation in real-time. As such, roslyn is just answering the questions of that feature. If you would prefer less CPU being used, then disabling real-time, computationally expensive questions is the way to go.

[sec]

Any guide on how to disable those CPU consuming things and to leave only intelisense enabled for ex.? Switching from VS 2019 to 2022 caused a massive spike of CPU usage and having those options enabled by default doesn't help...

[Mike-E-angelo]

If you would prefer less CPU being used, then disabling real-time, computationally expensive questions is the way to go.

I would feel better about your suggestion @CyrusNajmabadi if both steps 5 and steps 8 resulted in the same CPU utilization, but they do not. It works as expected the first time (step 5), but on the second (step 8) it is showing wildly different results.

As such, roslyn is just answering the questions of that feature

Is this perhaps due to Visual Studio asking the wrong question on the 2nd attempt? Or asking the right question wrongly, if that makes sense?

[CyrusNajmabadi]

I see the same utilization for both.

[CyrusNajmabadi]

Any guide on how to disable those CPU consuming things and to leave only intelisense enabled for ex.? Switching from VS 2019 to 2022 caused a massive spike of CPU usage and having those options enabled by default doesn't help...

@sec i have no idea if youi're running into the same issue as i have not seen any traces from you. If it is the same issue, you can disable codelens here:

You will likely have to restart. If you disable this and see low CPU usage, then this was hte issue.

If you see high CPU usage after the restart still, then we'll need an actual trace to determine which feature is at fault.

[sec]

I have codelens already disabled, I will try to make trace when I will have some more time, more stable way to reproduce the issue.

[Mike-E-angelo]

I see the same utilization for both.

Perhaps try again? I have also seen this occur after several attempts where the first or even the 2nd may not take, but it does eventually happen.

In any case, you do now have a recording (as you have been requesting) of what is occurring on my machine, and it shows nominal/expected CPU on the first attempt and much different results on the 2nd. I am not making this up. :)

[CyrusNajmabadi]

Perhaps try again? I have also seen this occur after several attempts where the first or even the 2nd may not take, but it does eventually happen.

I've tried about 10 times now.

[CyrusNajmabadi]

@Mike-E-angelo does this issue go away if you disable code-lens?

[CyrusNajmabadi]

I have codelens already disabled, I will try to make trace when I will have some more time, more stable way to reproduce the issue.

Yeah, we'd need a trace then. This process is effectively a server that does many things. You may see the same symptoms for very different reasons. If you have codelens disabled, then honestly we should have a second issue for what you're seeing @sec so that it can get tracked and addressed independently.

[Mike-E-angelo]

I've tried about 10 times now.

I guess we have my recordings to ascertain what is happening here. Or if someone else is able to reproduce it we can determine what is the common environmental factor that is clearly missing on your machine.

@Mike-E-angelo does this issue go away if you disable code-lens?

Indeed as discussed this does go away but then I no longer have Code Lens. :) Additionally there is still the gotcha here of it working as expected the first time but not on the second. To me that implies a gremlin lurking about that is probably leading to the problems others are reporting.

[sharwell]

@sec Note that excessive CPU for CodeLens is highly specific to individual projects. In general, even larger projects like Roslyn can use this feature without observable negative impact. It's not really possible to suggest customers facing "high CPU" disable it, since it only affects a tiny minority of all reports with that title we've seen in the entire VS 2022 release (definitely <5%).

The best way to get an issue like this investigated is to file a feedback using these steps: https://aka.ms/reportPerf

[Mike-E-angelo]

@sec Note that excessive

Thank you for tagging/clarifying your intended audience @sharwell because you were about to get a lengthy reply from me. 😆

That stated, I did re-read the instructions you provided and it mentions using a new/separate instance of Visual Studio which I did not observe with this morning's recording. Please let me know if it's preferred to have one made from a separate Visual Studio instance and I will make another for you.

Also note that I can further provide a PerfView recording of this issue from my machine if needed. Let me know if that is necessary and/or how I can further assist in diagnosing this issue for you.

[CyrusNajmabadi]

@Mike-E-angelo do you use new(...) a fair amount in your projects? e.g. "implicit object creation"? Could you do a potential find-in-files search to tell me how many docs contain at least one new( hit, and compare that to the total number of docs you have in your solution?

[Mike-E-angelo]

Thank you for further looking into this @CyrusNajmabadi. For awareness and clarity here, please note that the solution I am using to report this issue is open source and freely available for your use.

Along those lines, can you please verify that you are using this same solution to attempt to reproduce this issue on your side or are you using a different one altogether? That might explain the reason we are running into different results on our different machines.

Establishing baselines aside, note that a quick search for new ( in this solution resulted in 50 hits in 40 files, and new( resulted in 531 hits in 330 files, for a total of 581 hits in 370 out of 2,682 files. So roughly ~13.8% of the files if I am eyeballing this correctly.

Please do let me know if there is any further information that I can provide and I will do my best to assist you.

[CyrusNajmabadi]

yeah, i'm seeing very high CPU needed to do the find-refs because we're looking for references to your constructors, and we have to check all of those potential matches because we don't know what they actually mean and can't filter out enough of them based on their syntax alone.

[CyrusNajmabadi]

Along those lines, can you please verify that you are using this same solution to attempt to reproduce this issue on your side or are you using a different one altogether?

I'm using yours, following the instructions you gave in the OP about which files to open and what to repeat :)

[Mike-E-angelo]

yeah, i'm seeing very high CPU needed to do the find-refs because we're looking for references to your constructors

Seems like this is working as expected with the first pass, but not the second. That is, the only time high CPU usage is on the 2nd adding of the files, hence the riddle here. 🤔

[sharwell]

@CyrusNajmabadi Can we add a syntax-based type hint to FAR for new() that appears in an initializer where the type name of the target type is available on the left hand side? This would bring performance parity for new() at least in the cases where we suggest target-typed new via an analyzer.

[CyrusNajmabadi]

@sharwell we could. However while that would get us better true-positive matches. It wouldn't help with the false-positive case. Effectively (and need to discuss with @jcouv to make sure of this) practically any downstream new(...) could be ina context where it is referring to the type in question. e.g. Foo(new(...)). We'd have to still check this to see if it possibly was a reference to the constructor in question.

The only ways around this are:

1. store some more syntactic information to hope to get less false positives. for example, storing the argument-count at the use site. this will only be beneficial though if the file the new(...) is in doesn't have other new(...) calls that run the gamut of the common arg counts (e.g. 0-4 args).
2. do an initial pass first to see where the containing type is used e.g. Bar in void Foo(Bar bar), then use that information and syntax to then try to filter down to documents that have both a usage of Foo and new(...) in it. However, this sort of 'combined knowledge' approach is something we have no precedence for, and it would require us to enumerate all possible contextual locations and ensure that all of them have something else we can definitively search for. I actually think this is promising, but it's def a large departure from how Find-Refs works today.

@jcouv could you actually enumerate all the scenarios in teh compiler hwere new(...) is allowed (e.g. where hte compiler can infer the type for it?). If you can, i can start assessing if all those scenarios would have a secondary indicator that we can use to drive this.

[xAstroBoy]

Can confirm, i have the same bug with VS 2022 Enterprise.

[CyrusNajmabadi]

@xAstroBoy if you have the same issue, you can disable codelens to make things better.

[Mike-E-angelo]

you can disable codelens to make things better

debatable :P

[CyrusNajmabadi]

@Mike-E-angelo The issue we see in your trace is CodeLens. And it appears from your above convo that disabling codelens addressed the issue. So i'm not sure what's debatable here.

If you are still seeing issues even with codelens disabled, we will need a trace to find out what the issue is.

[Mike-E-angelo]

I understand that @CyrusNajmabadi, and I appreciate the confirmation. However, recommending users to turn off a well-utilized, valuable, and important feature to "workaround" an issue is not exactly making things better. Note that if I had followed this advice I would have been without CodeLens for nearly six months and counting now. This is a harrowing thought for me as I use CodeLens about as much as I do R#, so I leave it on and endure the grief instead.

We can uninstall Visual Studio altogether and it will technically fix a bunch of problems, too. :P I don't think that makes things better at all.

[CyrusNajmabadi]

his is a harrowing thought for me as I use CodeLens about as much as I do R#, so I leave it on and endure the grief instead.

I'm not sure what to tell you. The issue is non trivial. :)

CodeLens works by getting reference counts to the items you have in the file. That is extremely expensive as we have to go find all those references. As such, the CPU utilization is not abnormally high, it is in line with what is needed to actually find the references.

If you want low CPU you can disable this feature. If you want the functionality of the feature, then part of that is allowing our out of process server the CPU and memory to actually compute the results it needs, and it is no longer abnormal.

[CyrusNajmabadi]

@Mike-E-angelo if you can collect a trace again with the latest vs preview bits, that would be helpful. Thanks! :)