search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.3k stars 9.97k forks source link

High memory usage on an idle web app #53933

Closed georgiosd closed 4 days ago

georgiosd commented 7 months ago

Is there an existing issue for this?

Describe the bug

We are seeing high memory usage on fairly idle web apps (2-3 users, dev environment).

On a fresh start, mem usage is ~250MB. After some time (I haven't figured out yet how long this takes), the usage climbs to 2-4GB.

These apps are hosted on a cluster of Debian 11 servers.

Expected Behavior

I would have expected than an idle app stays well below 500MB of memory usage.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

6.0.26

Anything else?

No response

gfoidl commented 7 months ago

Can you share some more information about the app and the infrastructure to make the issue actionable? Ideally with a self-containing repro.

Did you try dotnet-counters to see if GCs are happening or not?

georgiosd commented 7 months ago

Hey @gfoidl thanks for picking this up on a Saturday!

I don't think I can provide a repro since I have no idea what could be causing this and the whole app won't exactly be minimal.

It's a "standard" ASP.NET app with several controllers, health checks etc. A couple of long running tasks running as hosted services. Nothing fancy like image processing or anything like that.

I've isolated the app in another box so I can install the SDK. Is this the command I should be using to collect counters?

dotnet-counters collect --process-id <pid> --refresh-interval 3 --format csv

This is the output for the GC counters only, albeit for a short interval so lmk how long I should be logging this for:

2/10/2024 6:52:47 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:52:47 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:47 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:47 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:47 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:52:47 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:52:47 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:52:50 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:52:50 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:50 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:50 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:50 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:52:50 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:52:50 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:52:53 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:52:53 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:53 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:53 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:53 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:52:53 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:52:53 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:52:56 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:52:56 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:56 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:56 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:56 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:52:56 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:52:56 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:52:59 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:52:59 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:59 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:59 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:52:59 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:52:59 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:52:59 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:02 PM,System.Runtime,GC Heap Size (MB),Metric,144
2/10/2024 6:53:02 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:02 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:02 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:02 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:02 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:02 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:05 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:05 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:05 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:05 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:05 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:05 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:05 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:08 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:08 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:08 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:08 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:08 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:08 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:08 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:11 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:11 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:11 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:11 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:11 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:11 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:11 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:14 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:14 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:14 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:14 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:14 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:14 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:14 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:17 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:17 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:17 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:17 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:17 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:17 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:17 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:20 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:20 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:20 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:20 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:20 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:20 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:20 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:23 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:23 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:23 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:23 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:23 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:23 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:23 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:26 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:26 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:26 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:26 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:26 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:26 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:26 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:29 PM,System.Runtime,GC Heap Size (MB),Metric,145
2/10/2024 6:53:29 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:29 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:29 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:29 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:29 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:29 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:32 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:32 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:32 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:32 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:32 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:32 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:32 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:35 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:35 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:35 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:35 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:35 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:35 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:35 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:38 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:38 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:38 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:38 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:38 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:38 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:38 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:41 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:41 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:41 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:41 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:41 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:41 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:41 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:44 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:44 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:44 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:44 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:44 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:44 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:44 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:47 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:47 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:47 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:47 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:47 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:47 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:47 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:50 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:50 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:50 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:50 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:50 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:50 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:50 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:53 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:53 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:53 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:53 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:53 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:53 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:53 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:56 PM,System.Runtime,GC Heap Size (MB),Metric,146
2/10/2024 6:53:56 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:56 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:56 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:56 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:56 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:56 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:53:59 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:53:59 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:59 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:59 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:53:59 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:53:59 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:53:59 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:02 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:02 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:02 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:02 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:02 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:02 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:02 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:05 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:05 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:05 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:05 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:05 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:05 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:05 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:08 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:08 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:08 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:08 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:08 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:08 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:08 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:11 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:11 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:11 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:11 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:11 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:11 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:11 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:14 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:14 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:14 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:14 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:14 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:14 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:14 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:17 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:17 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:17 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:17 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:17 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:17 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:17 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:20 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:20 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:20 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:20 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:20 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:20 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:20 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:23 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:23 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:23 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:23 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:23 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:23 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:23 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:26 PM,System.Runtime,GC Heap Size (MB),Metric,147
2/10/2024 6:54:26 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:26 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:26 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:26 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:26 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:26 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:29 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:29 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:29 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:29 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:29 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:29 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:29 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:32 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:32 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:32 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:32 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:32 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:32 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:32 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:35 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:35 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:35 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:35 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:35 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:35 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:35 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:38 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:38 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:38 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:38 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:38 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:38 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:38 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:41 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:41 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:41 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:41 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:41 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:41 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:41 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:44 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:44 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:44 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:44 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:44 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:44 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:44 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:47 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:47 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:47 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:47 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:47 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:47 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:47 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:50 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:50 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:50 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:50 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:50 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:50 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:50 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:53 PM,System.Runtime,GC Heap Size (MB),Metric,148
2/10/2024 6:54:53 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:53 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:53 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:53 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:53 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:53 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:56 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:54:56 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:56 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:56 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:56 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:56 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:56 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:54:59 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:54:59 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:59 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:59 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:54:59 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:54:59 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:54:59 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:55:02 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:55:02 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:02 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:02 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:02 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:55:02 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:55:02 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:55:05 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:55:05 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:05 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:05 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:05 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:55:05 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:55:05 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:55:08 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:55:08 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:08 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:08 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:08 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:55:08 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:55:08 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:55:11 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:55:11 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:11 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:11 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:11 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:55:11 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:55:11 PM,System.Runtime,% Time in GC since last GC (%),Metric,0
2/10/2024 6:55:14 PM,System.Runtime,GC Heap Size (MB),Metric,149
2/10/2024 6:55:14 PM,System.Runtime,Gen 0 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:14 PM,System.Runtime,Gen 1 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:14 PM,System.Runtime,Gen 2 GC Count (Count / 3 sec),Rate,0
2/10/2024 6:55:14 PM,System.Runtime,GC Fragmentation (%),Metric,5.904406835939009
2/10/2024 6:55:14 PM,System.Runtime,GC Committed Bytes (MB),Metric,215
2/10/2024 6:55:14 PM,System.Runtime,% Time in GC since last GC (%),Metric,0

PS: While the app doesn't get a lot of traffic, there is a LB that hits the health check every second or so. In case that matters. I haven't added any manual health checks.

gfoidl commented 7 months ago

Debug a memory leak in .NET is a tutorial for (one way of) investigating such issue. Please give this a try, and also have a look at dotnet-trace.

I'd run just dotnet-counters monitor (with the process id or process name, of course) for a couple of minutes to see if a GC is happening or not. If no GC is done, then it may be normal GC-behavior as there was no reason to trigger one (e.g. enough memory available, etc.). If there are GCs happening, then it may be a leak which you can investigate via the other options provides in the linked tutorial.

When you have troubles investigating a memory dump we can help out (but it's always hard when one doesn't know the app).

For more in-depth infos about GC and it's investigations see https://github.com/Maoni0/mem-doc

there is a LB that hits the health check every second or so

Can you run the app without the LB probing the health -- just to try to exclude a potential reason. Also if you can run the app on Windows and see the same behavior maybe it's worth running the app under PerfView to get more insights whats going on.

georgiosd commented 7 months ago

Okay, I'll see if I can replicate the issue locally and go from there. I will revert with any findings.

georgiosd commented 7 months ago

Still on this. Will revert with my findings.

On Mon, 26 Feb 2024 at 03:16, dotnet-policy-service[bot] < @.***> wrote:

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies https://aka.ms/aspnet/issue-policies for more information.

— Reply to this email directly, view it on GitHub https://github.com/dotnet/aspnetcore/issues/53933#issuecomment-1963154431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMYG5X5CD3JNW2UQBFDVZLYVPO6JAVCNFSM6AAAAABDCQENH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGE2TINBTGE . You are receiving this because you were mentioned.Message ID: @.***>

georgiosd commented 4 days ago

Closing this as I've failed to work on it for a few months. Will reopen if I get any evidence.