Asp.net Core not Collecting Garbage

[zorthgo]

I can't understand why Asp.net core doesn't seem to be collecting garbage. Last week I let a web service run for a few of days, and my memory usage reached 20GB. GC doesn't seem to be working properly. So to test this I wrote a very simple web method that return a large collection of strings. The application started off using only 124MB, but with each time I called the web method, the memory usage kept getting higher and higher until it reached 411MB. It would have gone higher if I had kept calling the web method. But I decided to stop testing.

My test code was this:

`

[HttpGet] public async Task<IEnumerable> TestGC() { const string message = "TEST"; return Enumerable.Repeat(message, 100000000); } `

Although I might be overlooking something... To my understanding, the memory usage should not be increasing with every call to this method. After the object is created and sent to the UI, the memory should have been freed.

As you can see from the screenshot below, even after the GC got called, the memory was not released.

Thanks for the help! Jay

[wanton7]

Or from @RemyArmstro He talks about different sized dictionaries. I checked corefx and Dictionary will allocate array or these

private struct Entry
{
  public int hashCode;    // Lower 31 bits of hash code, -1 if unused
  public int next;        // Index of next entry, -1 if last
  public TKey key;           // Key of entry
  public TValue value;         // Value of entry
}

85000 byte allocation will cause LOH allocation so Dictionary with capacity of 5313 entries of int key and int value will cause LOH allocation. Capacity is not same as number or entries, capacity seems to be expanded by primes, check private Dictionary's private Resize method. Each struct could have extra allocation plus memory padding so even lower entries could cause LOH allocation.

Dictionary implementation details Dictionary.cs

Edit: fixed url

[Jabronironi]

@wanton7 Thank you. I think we realize what the issue is now. It is just a bummer there is no great + simple solution to this. It basically comes down to being more aware of it and adjusting how you write code. The downside is that this unmanaged memory starts to feel a bit more managed. :( In the end, we may only have a few areas that truly breach this allocation limit, but one of the areas is pretty core to our app, so we see it a lot currently. I think we just need to rethink that piece, monitor, and try to ferret out any other areas we notice this creep. Thanks again!

[wanton7]

We actually have similar situation soon and we need to create chunked IList<T> implementation. I'm going to use some size for chunks that is bitshiftable so I can just use bitshift and mask for indexing.

I would like to know which one is more beneficial for GC, bigger chunk or smaller chunk? From sizes between 1KB and 64KB. Smaller chunks mean more references for GC but I would guess that bigger chunk might be worse for compacting and fragmentation.

[Maoni0]

your understanding is exactly correct - I would go with sizes not too large; probably try 4k/8k.

[wanton7]

@Maoni0 thank you!

I chose 4KB, so that we don't get any nasty surprises if we ever run our code under Mono. Found out by reading http://www.mono-project.com/docs/advanced/garbage-collector/sgen/working-with-sgen/ that LOH threshold is only 8000 bytes under Mono.

[atpyk]

Is there any progress for this issue?

[chrisaliotta]

We are observing the same issue where process memory continues to grow despite the heap size remaining the same (or decreasing).

[Eilon]

@sebastienros can you take another look at this? Maybe we can provide some detailed instructions to help people investigate their scenarios further?

[chrisaliotta]

Here are some items to consider with our situation:

• We're only returning a Dictionary<int, int> object with 1000 integer values stored. Nothing more than that.
• We converted our project from .NET Core 2.0 --> 2.1
• We are seeing this issue in MS Visual Studio 2017

Code as follows:

    public async Task<IActionResult> SampleAction()
    {
        var list = new Dictionary<int, int>();
        for (int n = 0; n < 1000; n++) list.Add(n, n);
        return Ok(list);
    }

To reproduce you must simulate some form of moderate load. We just clicked rapidly using Postman and could observe this behavior. Once we stopped simulating load, we watched the heap size decrease yet the process memory remained the same (even when we forced GC).

[beef3333]

I am seeing this as well in one of my projects, but I also can re-create it in a brand new .net core API targeting .Net Core 2.1 (SDK 2.1.302).

I attached the sample API project that I created using Visual Studio 15.8.0 Preview 4. To show the memory increasing I had a .net core console app hit the default values GET endpoint every half second to get the 2 strings. The process memory usage is slow but in a project that returns more data this can grow quickly.

WebApplication1.zip

[chrisaliotta]

I found this post on stack exchange:

https://stackoverflow.com/questions/48301031/why-doesnt-garbage-collector-in-net-core-2-0-free-all-memory

Has anyone profiled their applications in release mode to see if this behavior is present? I'll give this is a shot today and see if the issue persists.

Edit: I tried profiling in Release mode and the issue still persists. I even force a GC to see if that will have any effect.

[Eilon]

@chrisaliotta thanks for linking to that post, I was unaware of that behavior. It would indeed be interesting to see if that explains what people are seeing.

[sebastienros]

@Eilon @chrisaliotta Thanks but this is not related to this discussion. .NET not releasing memory in debug mode is a well known behavior and this is why we only measure memory consumption (and potential leaks) in release mode. Also even on release mode you'll see the memory increase to an extent overtime because of the Server GC mode. So this example doesn't prove anything for two different reasons.

[chrisaliotta]

@sebastienros So is the behavior that @beef3333 and I are observing consistent with what is to be expected? Namely, where Private Bytes remains elevated despite heap size decreasing? If so, it seems strange to me that each incremental request would continue to cause the private bytes to grow despite there being free heap space.

[sebastienros]

In Debug mode yes. So please try to use Release mode, and run your stress for a long time. If the memory increases indefinitely then there is a memory leak. If the memory is recycled (even if it takes some significant amount of memory) then all is fine in your case.

[beef3333]

I just tested it again using the same project I attached in Release mode and I am seeing the same exact behavior.

[sebastienros]

I will test your application then, thanks.

[sebastienros]

I ran the application provided by @beef3333 locally for 2 hours, with a rate of 5K RPS, and the memory is stable (variance by 1MB overall, at 400MB on a machine with 32GB). The GC is correctly called regularly. I also inspected multiple dumps overtime and the various instances get created and collected as expected. I am using 2.1.1.

[chrisaliotta]

@sebastienros Thank you for looking into this. I think the takeaway in all of this is that:

• Memory management will behave differently for a .NET Core web application versus your run-of-the-mill desktop application.
• Developers should focus on average memory consumption as a function requests per second (RPS) over a period of time.
• Growth in private bytes may not always be indicative of a memory leaks.

Correct me if I'm wrong, but it appears that .NET Core will grow allocated memory based on average requests in order to ensure the fastest response time? If true, would it be safe to assume that it is likely not to release this allocated memory until the Application Pool resets -- or will it release this allocated memory over time if RPS decreases?

[atpyk]

Same issue. I have an asp.net webapi backend service. The stacks are asp.net mvc, autofac, automapper, castle.dynamicproxy, entity framework core. All memory will be eaten, then service crash.

Version is 2.1.0

[wanton7]

@atpyk Update to 2.1.1. If that doesn't help you should really profile what's keeping that memory. I've used https://www.jetbrains.com/dotmemory/ but there are probably other tools that can do this as well. It can show what's actually allocated in Large Object Heap (LOH).

Are you running in 32bit mode? Because Large Object Heap allocations (bigger than ~85000 bytes) can cause out of memory exceptions in 32bit mode due to fragmentation. You can get over this limit very easily using Dictionary<T, T>. Check this comment https://github.com/aspnet/Home/issues/1976#issuecomment-393833505

If you are running your code in full .Net Framework default behavior is running any cpu code in 32bit mode. You need to uncheck Prefer 32bit from project settings or set some registry setting in your servers registry to default to 64bit.

[atpyk]

@wanton7 Many thanks. I will try your solutions.

[atpyk]

I updated to 2.1.2 and deploy on Microsoft Azure webapp with win-x64, but no effect. @wanton7

[sebastienros]

@atpyk please create a memory snapshot (dump_ and analyze it (Visual Studio, MemoScope) to see what object are taking all the memory, or just increasing in count. You can also take two and compare them overtime.

[SergeySagan]

@sabastienros I do believe that enough people have raised a concern over this that you/MS should actually start analyzing this yourself. Perhaps this is working as you intended, but then the design is flawed.

Our apps are eventually running out of memory and crashing, and this all running in a production Azure environment. This is not acceptable.

[wanton7]

@atpyk then it sounds like a memory leak. Profile your memory to see what is keeping that memory around like @sebastienros said.

One question, are you even using ASP.NET Core? I read you first comment again and you mention ASP.NET MVC. ASP.NET MVC and ASP.NET Core are two completely different products. These issues and this repo for ASP..NET Core.

Edit: Just from version numbers sounds like you are using ASP.NET Core, but wanted to make sure.

[atpyk]

We use .net core MVC. @wanton7 I am doing analyze the memory. Maybe the Castle Dynamic Proxy lead to memory leak.

[wanton7]

@Serjster are your programs running in 32bit .NET Core and crashing with out of memory exceptions? If your code does lot of LOH allocations then memory fragmentation is probably the reason. If you are running in 32bit environment you have two choices, fix your code to avoid LOH or switch to 64bit environment.

I'm not very familiar with Azure but after little bit googling found this https://blogs.msdn.microsoft.com/webdev/2018/01/09/64-bit-asp-net-core-on-azure-app-service/

[sebastienros]

@Serjster I believe not all reports here are equal, and prefer to check the validity of each different case. Something like "my app has a memory leak" doesn't mean it's because of the framework, so I prefer to ensure each case it an actual one.

Taking your case for instance, "I believe" that I answered the reason why your memory is increasing. Have you been able to fix it after my explanation? Or it didn't solve the issue and in this case can you give an application I can run locally to reproduce the issue?

[Jabronironi]

@sebastienros What we ended up finding was that memory allocation just kept increasing, even though memory consumption did not. I know ASP.NET core is doing some heuristics to tell it that it should grab more, but it seemed to be constantly allocating more and more on each request. It almost appears to greedy to me in this regard, but I could be wrong.

Either way, I think @Serjster point is that this thread keeps growing because there is clearly some confusion here. In the old ASP.NET land (before core 1 I believe), we did not need see this behavior (at least not at this magnitude. It may not really be a bug/issue, but it is definitely something causing a lot of people to pose the same question over and over again.

It would be nice if there was an official article that really address this thread from top to bottom, instead of it going in circles as it has been. Hope that helps clarify.

[chrisaliotta]

It would be nice if there was an official article that really address this thread from top to bottom, instead of it going in circles as it has been. Hope that helps clarify.

I second that. It would be good for us to know why .NET Core 2.1 is more "opportunistic" when it comes to memory management than prior versions.

[Maoni0]

@sebastienros can we please have a summary of the issues here? there are 81 comments - I'm under the impression they are not all about the issues (although I have not read them carefully at all so I might be mistaken). if so can we list all the distinct issues here and see if we have repros for each of them? there are enough folks who have mentioned the memory increase I think it justifies us to get repros and figure out whether these are general issues or not.

[sebastienros]

Sure. I am currently trying to identify all these threads and see what the status is for each of them. I will eventually close this issue and reopen more specific ones that focus on each report, because this single thread isn't sustainable anymore. I'll link them here for those on this thread that want to follow them.

In parallel I will work on a document that lists all the recommendations, the known memory issues (LOB, HttpClient, ...) and the ways to analyze and report these issues.

[sebastienros]

Just to assure you we do care about this issue and in order to detect memory leaks preemptively, for the past 6 months we have been running an ASP.NET applications continuously on Azure, on both Linux and Windows, for 24h and 7 days. These tests fetch the latest ASP.NET source on every iteration (daily or weekly). We measure RPS, latency, CPU and memory usage. The application uses EF Core, MVC and Razor, and is sustained to 50% CPU to simulate a significant load.

You can see the results publicly here on this site (browse to find the daily report): https://msit.powerbi.com/view?r=eyJrIjoiYTZjMTk3YjEtMzQ3Yi00NTI5LTg5ZDItNmUyMGRlOTkwMGRlIiwidCI6IjcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0NyIsImMiOjV9&pageName=ReportSectioneeff188c61c9c6e3a798

This has allowed us to fix some problems by the past, and to be confident that there is no fundamental leaks in the system right now. But it's nowhere close to using all the components that we ship, and there might be issues that we need to identify. Providing dumps and applications that reproduce the issues are the main ways you can help us.

[Jabronironi]

@sebastienros Thank you for the updates. I know in our case the issue was more about a new "greedy memory allocation" concern, which we original mistook as a memory leak. I am not even sure there is an issue now, it could just be that the new optimization heuristics are a lot more aggressive. Not sure... but I think you are on the right track with really assessing this thread and coming up with some consolidated explanations/summaries on what people are seeing/misunderstanding. Good luck!

[sebastienros]

All individual reports have been isolated and will get individual follow-ups, and be closed if they are already solved. Feel free to subscribe to these to be kept in the loop.

[PureKrome]

In parallel I will work on a document that lists all the recommendations, the known memory issues (LOB, HttpClient, ...) and the ways to analyze and report these issues.

This is a huge +1 from me. I feel that one of the biggest issues here is how hard it 'feels' to gather information, to then try and help determine what is the problem. Having some great docs that can allow us to follow some instructions to both (i) gather, (ii) attempt a self-diagnose and (iii) post our dumps/findings in a way that is efficient for the MS team can really help both sides of the fence.

If we (the developers) can be better-enabled to diagnose and/or provide information, then this is a win-win for all.

Thanks again for listening @sebastienros - very much appreciated, mate!

[alexiordan]

What do you think about situation in the picture below?

We run 4 WebApps inside the same Plan. Initial it was B1, scaled up to S2 and memory kept growing until I set to the hungry webapp csproj:

<ServerGarbageCollection>false</ServerGarbageCollection>

and also disable Application Insights

1. I believe that since memory could be kept under control with setting above, there is no memory lea. Correct?
2. Is the presented behavior normal?

[oncicaradupopovici]

Same situation here as @alexiordan, we had a .net core 2.1 console, that ran some IHostedServices hosted in Kube, FROM microsoft/dotnet:2.1-runtime AS base. We wanted to enable HealthChecks so we added asp.net with just the HealthChecks middleware and changed the base image to microsoft/dotnet:2.1-aspnetcore-runtime. The result was OOM. We have managed to stabilize the memory allocation by adding false in csproj.

Our analysis shown that in an asp.net app The GC collects less frequently, also the Finalizer Queue is traversed less frequently.

Also, if we forced the GC to collect and traverse the finalizer queue by adding the following in our pipeline,

System.GC.Collect(); System.GC.WaitForPendingFinalizers(); System.GC.Collect();

the memory allocation remained stable.

[dguerin]

What do you think about situation in the picture below?

We run 4 WebApps inside the same Plan. Initial it was B1, scaled up to S2 and memory kept growing until I set to the hungry webapp csproj:

<ServerGarbageCollection>false</ServerGarbageCollection>

and also disable Application Insights

1. I believe that since memory could be kept under control with setting above, there is no memory lea. Correct?
2. Is the presented behavior normal?

Hey @alexiordan

We're seeing a very similar memory profile when using AI (web app on net core 2.1) as well, have you progressed any further on solving this? Obviously we want to keep AI in the apps.

[kgrosvenor]

It is weird that the usage grows on each request, but setting the above to false seems to stop it for me? weird because you'd think true would be the value that enables it, but it seem seems the other way around...

[sebastienros]

I forgot to mention that soon after announcing my intention to write an article about the issues described in this thread I actually did it. You can see it here: https://github.com/sebastienros/memoryleak

It comes with a small application that renders the patterns on a chart, live.

[benaadams]

but setting the above to false seems to stop it for me? weird because you'd think true would be the value that enables it, but it seem seems the other way around...

Client garbage collection (optimised for memory sharing with many apps and keeping memory free) is more aggressive than Server garbage collection (optimised for throughput and concurrency).

[kgrosvenor]

Setting SGC to false, my asp.net core api fell from 150mb to 48mb, and didn't grow on each request after that. So in reality is this the best setting for production, for now?

[matteocontrini]

@kgrosvenor actually, it depends. Quoting from the excellent @sebastienros article:

On a typical web server environment the CPU resource is more critical than memory, hence using the Server GC is better suited. However, some server scenarios might be more adapted for a Workstation GC, for instance on a high density hosting several web application where memory becomes a scarce resource.

[kgrosvenor]

Thank you that's very handy, I will bear in mind - will follow this thread for any more updates, that being said I'm absolutely loving asp.net core :)

[cemdev]

Also suffer from this with a .net core 2.1 console app. constant memory growth. have set docker containers at a low maximum so it hits it and restarts, which is working okay, but it's ugly.

[flo8]

Any news on this side? We also have the same behavior in ASP.Net Core v2.1. From what I can see in the commit https://github.com/aspnet/AspNetCore/commit/659fa967a1900653f7a82f02624c7c7995a3b786 it seems there was a memory management problem that is going to be fixed in v3.0?

[benaadams]

@flo8 have you tried upgrading to 2.2 and checked the behaviour? https://dotnet.microsoft.com/download

[dre99gsx]

Running the latest 2.2 and also have this issue - merely creating a list of 1 million ints and returning an emptyResult() will increase my heap a few hundred MB every request until I run out of memory. Setting ServerGarbageCollection to false cured it, although it doesn't seem like the correct fix...