search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.47k stars 10.03k forks source link

Memory leak: Property caches are not getting cleaned up when using AssemblyParts. #21744

Open mrukas opened 4 years ago

mrukas commented 4 years ago

Describe the bug

When you load an assembly in a new AssemblyLoadContext and add it dynamically via an AssemblyPart to the ApplicationPartManager new entries are getting added to the internal property caches. These entries do not get removed when the AssemblyPart is removed and the assembly is unloaded.

These are the caches that i am talking about: PropertiesCache VisiblePropertiesCache.

Because same types loaded in a different AssemblyLoadContext are not reference equal more and more entries are getting added.

It would be counterproductive to clear both dictionaries. At least the entries belonging to the unloaded assembly should be removed. Although i guess that the performance impact of clearing the whole cache wouldn't be high.

If you load and unload dlls multiple times you can clearly see the issue.

Without clearing the property caches: Without cache clear

With clearing the property caches: with cache clear

To Reproduce

I added a repo that can be used for investigation: Link If line 111 in AssemblyController is commented you experience a memory leak. If you trigger this method multiple times you can see in the returned result, that for every load an entry gets added to the PropertiesCache. If you comment this line and call this method in a production build you can see a significant increase in memory consumption. I did not experience an increase of the number of entries in the VisiblePropertiesCache, but I'm pretty sure both caches should be cleaned up if they contain an entry for an unloaded assembly.

Further technical details

.NET Core SDK (gemäß "global.json"): Version: 3.1.100 Commit: cd82f021f4

Laufzeitumgebung: OS Name: Windows OS Version: 10.0.18363 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\3.1.100\

Host (useful for support): Version: 3.1.1 Commit: a1388f194c

.NET Core SDKs installed: 1.0.4 [C:\Program Files\dotnet\sdk] 1.1.0 [C:\Program Files\dotnet\sdk] 2.1.2 [C:\Program Files\dotnet\sdk] 2.1.4 [C:\Program Files\dotnet\sdk] 2.1.101 [C:\Program Files\dotnet\sdk] 2.1.104 [C:\Program Files\dotnet\sdk] 2.1.105 [C:\Program Files\dotnet\sdk] 2.1.201 [C:\Program Files\dotnet\sdk] 2.1.202 [C:\Program Files\dotnet\sdk] 2.1.300 [C:\Program Files\dotnet\sdk] 2.1.301 [C:\Program Files\dotnet\sdk] 2.1.302 [C:\Program Files\dotnet\sdk] 2.1.400 [C:\Program Files\dotnet\sdk] 2.1.402 [C:\Program Files\dotnet\sdk] 2.1.403 [C:\Program Files\dotnet\sdk] 2.1.504 [C:\Program Files\dotnet\sdk] 2.2.106 [C:\Program Files\dotnet\sdk] 2.2.202 [C:\Program Files\dotnet\sdk] 3.1.100-preview3-014645 [C:\Program Files\dotnet\sdk] 3.1.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.0-preview3.19555.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.0-preview3.19553.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.0-preview3.19553.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

davidfowl commented 4 years ago

I’m sure there are more things that don’t get cleared when assemblies get unloaded. This isn’t something we’ve designed for so I would expect this to be the tip of the iceberg.

If you are willing to send PRs to solve the issues as you did them, that would be great. Otherwise, we’d need to look at this all up but, that’s super unlikely because it’s low priority

mrukas commented 4 years ago

It could really be that there are more things that doesn't get cleaned up. But when analyzing the memory usage this seemed to be the most prominent error. I will try to further investigate the issue and try to provide a PR if I find the time.

davidfowl commented 4 years ago

cc @janvorli

mrukas commented 4 years ago

It appears that not only the property caches do net get cleared but also the cache of the ModelBinderFactory is growing with each loading and unloading. Link

brunolins16 commented 2 years ago

I am looking at this issue and what if instead of clear cache we do not cache the type at all when it was loaded from a different AssemblyLoadContext, since the issue came from the assembly been loaded in a different AssemblyLoadContext instead of the default one?

halter73 commented 2 years ago

Disabling the cache entirely for types loaded from non-default AssemblyLoadContexts seems like it would hurt perf pretty badly for affected scenarios. Could we just clear caches when the IActionDescriptorChangeProvider fires?

brunolins16 commented 2 years ago

@halter73 the perf regression is something I was concerned but I am not able to figure out yet.

I have been trying to understand where not caching the controller (loaded in the non-default context) could cause issues, but I was able to find it been called during the initial app request or when the collection is changed and notified by the ChangeToken that seems reasonable for me. Do you happen to know where else it could be affected because I am probably missing something?

gyankov commented 2 years ago

Has anyone been able to find all caches that reference the assembly?

ghost commented 2 years ago

Thanks for contacting us. We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.