Memory leak in application using IdentityServer

[kmmccafferty96]

Which version of Duende IdentityServer are you using? 6.3.5

Which version of .NET are you using? .NET 8 (issue was happening in .NET 7 as well)

Describe the bug

Not sure if it's a bug or not, but we have an application which uses Duende IdentityServer that seems to have a memory leak. We have the application deployed in kubernetes and noticed that the memory usage kept spiking to 20GB+. Upon running the application locally, it seems that hitting just the /connect/authorize endpoint continuously (and doing nothing else) will cause "unmanaged memory" to spike in a linear fashion. We are using AddInMemoryIdentityResources, AddInMemoryApiScopes, AddInMemoryApiResources, and AddInMemoryClients, which I thought could be the culprit, however it is now my understanding that these resources are loaded into memory just once and not on each request.

It's possible the leak is coming from somewhere else in the application, however it is strange that only spamming the /connect/authorize endpoint causes the spike.

Another thing to note, I am using Rider/dotMemory on Linux to do memory profiling for this app, and it apparently cannot see anything in "unmanaged memory".

To Reproduce

1. Setup a .NET application that uses Duende IdentityServer
2. Continuously call the /connect/authorize endpoint.
3. Observe memory spike

Expected behavior

Memory should be garbage collected properly after spamming the service.

Log output/exception with stacktrace

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. Is thrown if we put a limit on the memory, but otherwise it will keep growing.

Additional context

The application is using System.IdentityModel.Tokens.Jwt v7.5.1 and is not using Microsoft.ApplicationInsights.AspNetCore (from this comment https://github.com/DuendeSoftware/Support/issues/308#issuecomment-1302335972)

[AndersAbel]

We have had occasional reports of memory leaks in IdentityServer deployments in the past. In the cases where we have been able to find the root cause it has always been in the hosting application, outside of IdentityServer. There are cases (such as the linked #308 issue) where we didn't get all the way to a root cause, but those are rare. If I am to make a guess I would say that this is probably something with your deployment/hosting application. If it would be a general issue in IdentityServer we would have heard more about it.

But you can never be sure, it could be that you are running with a combination of settings that actually causes issues in IdentityServer. As far as I know IdentityServer only uses managed memory, so if it is indeed unmanaged memory it indicates that the issue is outside of IdentityServer.

To know for sure we would need some profiling data that indicates what kind of data is doing those allocations. Could it be logging? Could it be somewhere in the persisted grant store? Could it be the profile service implementation?

[RolandGuijt]

@kmmccafferty96 Did Anders' comment solve it for you? If so I'd like to close.

[kmmccafferty96]

@RolandGuijt sorry I have been busy with other things but from preliminary testing it looks like he is correct that it is not Duende. If I find that it is I will reopen. Thanks!