search

dotnet / diagnostics

This repository contains the source code for various .NET Core runtime diagnostic tools and documents.
MIT License
1.18k stars 354 forks source link

Is there a recommended process to follow for a dotnetcore memory dump in production? #2763

Closed joncoelloaccess closed 2 years ago

joncoelloaccess commented 2 years ago

I have an asp.net core 3.1 application running under IIS. It's had a fairly sudden increase in memory and I'd like to do a memory dump.

Is there a safe production process to follow? Its a load balanced environment to it would be straight forward enough to add and remove from the load balancer. There's no session state but we do use caching.

tommcdon commented 2 years ago

Hi @joncoelloaccess there are several techniques that can be used to diagnose memory issues in production. As with collecting any diagnostic artifact, whether it be a log file, performance trace, or memory dump, care should be taken to protect customer information. If you are using Azure, there are some great diagnostics built in. Please see Azure App Service Diagnostics.

Here is an enumeration of a few techniques for local diagnostics using "off the shelf" tools:

  1. Collecting .NET performance counters is usually the best place to start. Using dotnet-counters to determine what type of memory leak is occurring often helps to narrow down the nature of the leak - how fast it is occurring, which managed heap, etc... For example, if the memory leak is due to managed references in the GC heap, the GC memory counters will indicate the leak is occurring.
  2. Dotnet-gcdump collects the GC objects rooted in memory. Taking multiple GC dumps and comparing them will help narrow down the nature of the leak. Traces can be opened in Visual Studio or perfview. Visual Studio provides a "compare with baseline" feature useful to narrowing down memory leaks. Please see this article on examples of usage. Note that collecting a gcdump triggers a full Gen 2 garbage collection in the target process and can change the performance characteristics of your application. The duration of the GC pause experienced by the application is proportional to the size of the GC heap; applications with larger heaps will experience longer pauses.
  3. Collect dumps of the process. A memory dump may contain customer information, so care should be taken. Using multiple process dumps along with Visual Studios debug memory dump feature or windbg/SOS can be useful.
  4. If using Visual Studio remote or local debugging on the machine exhibiting the memory leak enables both collection of the diagnostic artifacts as well as analysis using the memory analysis tooling

Hope this helps!

joncoelloaccess commented 2 years ago

This is great, thanks v much :)