Process memory increase while heap size does not

[ssougnez]

Describe the bug

I'm observing an annoying behavior with a .NET Core 3.1 web application. The memory used by the IIS process gradually increases over time. Before posting, I did some investigations...

First, I did some measurements with the diagnostic tool of visual studio. Basically, when I do nothing on the application, it stays stable. However, I have a GraphQL route that fetch users from the database and when I call this one, the memory used by the process increases and never goes down. I looked for a memory leak in there but at the end of the day, it's simply a simple EFCore query. However, I took some memory snapshots to try and find the culprit and I noticed that while the memory used by the process increased from 476 Mo to more than 800 Mo, the heap size only increased from 38 Mo to 40 Mo

I also did some tests with dotMemory and the result are the same than the one in here: https://github.com/dotnet/aspnetcore/issues/1976.

I end up with more than 700 Mo of "Unmanaged memory", however, as you can see in the yellow and purple rectangle, the .NET memory usage increased from 42 Mo to 60 Mo while the total memory increased from 430 Mo to 840 Mo. The graphic also show that after the GC fired, the heap memory gets released but the total memory is not.

As you can imagine, this is pretty annoying as after 3 days, the IIS process of the machine is up to 3 Go and I have to recycle the app pool. Not something I like to do :-)

Further technical details

• ASP.NET Core 3.1

PS C:\WINDOWS\system32> dotnet --info .NET SDK (reflecting any global.json): Version: 5.0.200-preview.20614.14 Commit: 863605c8c3

Runtime Environment: OS Name: Windows OS Version: 10.0.17763 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\5.0.200-preview.20614.14\

Host (useful for support): Version: 5.0.1 Commit: b02e13abab

.NET SDKs installed: 2.1.519 [C:\Program Files\dotnet\sdk] 2.2.207 [C:\Program Files\dotnet\sdk] 3.1.101 [C:\Program Files\dotnet\sdk] 5.0.101 [C:\Program Files\dotnet\sdk] 5.0.200-preview.20614.14 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.All 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download

• Visual Studio 2019

[ssougnez]

After doing more investigation, I tried changing the GC to Workstation and it fixed the problem. I'll check the drawback of that solution but I guess this issue can be closed.

[pranavkm]

FYI @sebastienros

[davidfowl]

@ssougnez How many cores does your server have?

This doc can help understand managed memory a bit more https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md

[NewITCorp]

After doing more investigation, I tried changing the GC to Workstation and it fixed the problem. I'll check the drawback of that solution but I guess this issue can be closed.

Hi @ssougnez could you please elaborate more on changing GC mode? because I think the mode by default is already Workstation.

thanks

[shailesh64]

I have hosted one .net core 6.0 on iis.Our application fully depending on session. The worker process physically memory not increased. It reach the 3GB then again release the memory it clean the whole session. Please suggest me the solution.

[helm100]

I observe the same behavior: high memory usage and only a fraction of it is the heap ('private bytes' 787 MB, heap size 47 MB). Is this something to worry about? Could it point to a memory leak? Or is this memory only 'reserved' by the running application and easily released when needed by other processes? Maybe this is not the correct place to ask these questions, but I'm very curious about this topic and would like more insight.

[Applesauce314]

I am also observing this behavior, and since i am also dealing with a memory exhaustion issue, it would be helpful to know if this is normal behavior or if it is a bug or misconfiguration

[mitchdenny]

Note for folks posting on this old issue (that we should probably close). Memory related issues are very often scenario specific. In order for us to provide useful guidance and identify a bug we need a specific repro that we can analyze. @davidfowl provided a useful link above so that you can do the initial round of analysis yourself to rule out issues in your own code before we start looking at either ASP.NET Core or the .NET runtime.

@Applesauce314 if you believe you have a bug after doing some initial analysis yourself I recommend opening up a new issue with a repro as well as any diagnostics you've collected for your specific scenario. We can then try and help you figure out whether it is an issue on the ASP.NET/runtime side or something in your code.

I would also recommend reading the following as background:

https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-7.0

Note that server GC and workstation GC will result in different behaviors in terms of persistent memory utilization (this is to be expected).

[davidfowl]

I answered another one of these today https://github.com/dotnet/aspnetcore/issues/48641#issuecomment-1589414334

[helm100]

@mitchdenny agreed. However, I also believe memory related behavior is often not scenario specific, and related to the inner workings of dotnet, thus perhaps a relevant topic to discuss here.

It seems dotnet's memory usage confuses people (I've seen quite a few similar topics, never clearly answered) and I believe the key 'issue' might be that the Server GC tends to use more memory than people expect/like it to. However, thus far I've found no clear, official document with quantitative explanations. Some hints can be found here.

Hence I find your last comment very interesting! Could you elaborate, or provide a link? I also understand that the topic might depend much on the exact version of dotnet used, and therefore cannot be answered generally.

[davidfowl]

I pasted this earlier, but I can do it again:

https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md

[sa-es-ir]

@davidfowl Recently I had this memory issue and I fixed that by setting GC workstation I'm running on windows 10 and 4 cores (8 logical) and .NET 8.

Before:

After:

GC happens many more times under workstation as in the picture, we will use this API in docker container at the end, is that ok for using GC workstation?