Asp.net Core not Collecting Garbage (ronald7)

Copied from https://github.com/aspnet/Home/issues/1976

@ronald7 commented on Apr 10

Hi,

I think i am having the same kind of problem with a .Net Core Web API in production.

The application is running on Windows Server 2016 with .Net Core 2.0.3. The machine is a Hyper-V VM with 28 CPU cores and 24GB of RAM. If we don't recycle the IIS application pool often enough, we will eventually use all the available memory. When the application starts using a lot of memory (>=95% of total system memory) the CPU usage also increase strongly (from 2% to 70% sometimes). I am not sure sure if an OOM exception is triggered or not, we always recycle the pool before it happens (the maximum memory usage I have seen was 98% of memory used by dotnet.exe).

Analyzing a production memory dump with ".Net Memory Porfiler" (SciTech Software) here is what i found:

If this analyze is correct, about 95% of the memory is in "overhead > unused". Here is how this memory profiler editor is describing this category (on their forum): "Overhead->Unused" is memory committed by the .NET runtime for the managed heap. It is currently unused, but it is available for future instance allocations. There are lot of rules the runtime uses to decide whether to keep the committed memory or to release it to the OS. It depends on factors like the available memory, allocation patterns, the number of processors, whether the server GC is used, etc.

@jkotas I will apply your recommendations (Windows job object, and switching to workstation GC) and I will let you know the result. Please let me know if i can extract any other useful information from the production memory dumps i have.

Thanks

@sebastienros commented on Apr 10

@ronald7 Would any of you be able to share an app that shows the issue? If I could repro it we would be able to find the reason, or at least remove some code piece by piece and isolate a minimal repro.

@ronald7 commented on Apr 10

Hi @sebastienros

Unfortunately, I cannot share the app nor the memory dumps, but I will create a dummy application (with the same architecture and dependencies), run it on the same machine, if I can reproduce this behavior i will share this one with you. Please let me know if there any useful information I could extract for you from the memory dumps.

I have updated the GC mode from server to workstation on one production server, I will let you know in a few hours from now if it changes anything on the memory usage.

I also performed another test: we are running our application behind a load balancer, on 4 virtual machines. After removing one of the machine from the load balancer pool, the memory used by dotnet.exe did not decrease and remained at the same level even after 30 minutes. (However, the application was still processing a few requests: one request sent by SCOM on a dummy endpoint every 30 seconds). No memory was released and returned to the system.

Thank you

@ronald7 commented on Apr 11

Hi @sebastienros @Maoni0 ,

I ran our application with workstation GC mode for 12 hours but same result. I also recompiled the application with .Net 2.1 Preview 2 on a single production node for 1 hour, i will let you know the result, but for now the process is already using 2GB+ of RAM.

I have PerfView running on this same machine and i am collecting GC dumps, is there an email address where i could send you the OneDrive link, unfortunately i cannot share it directly in this thread.

If it can help i can also collect more metrics or GC logs. Thank you

@ronald7 commented on Apr 12

Hi @sebastienros @Maoni0 I just sent you an email with two PerfView gcdump and a VMMap file, I hope this can help. On my side I am still trying to reproduce this high memory usage behavior with a dummy application.

Thanks!

dotnet / aspnetcore

Asp.net Core not Collecting Garbage (ronald7) #3412