Issue with TimeoutException During Load Testing (Duende Identity Server 7 + Postgres OperationalStore)

[olhakhariv]

Which version of Duende BFF are you using? Duende Identity Server 7.0.7

Which version of .NET are you using? .NET 8.0

Describe the bug While conducting load testing with up to 50 simulated users (10 new users per second), we encounter a TimeoutException during the /connect/token request when the number of users increases

To Reproduce

1. Set up Duende Identity Server 7.0.7 with a PostgreSQL OperationalStore.
2. Simulate up to 50 users authenticating at a rate of 10 new users per second.
3. Observe the TimeoutException occurring in the /connect/token request after reaching the peak load.

Expected behavior We expected the system to handle the requests without timing out as the load increases

Log output/exception with stacktrace

contextType
Duende.IdentityServer.EntityFramework.DbContexts.PersistedGrantDbContext

error
System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
 ---> Npgsql.NpgsqlException (0x80004005): The operation has timed out
 ---> System.TimeoutException: The operation has timed out.
   at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
   at Npgsql.Util.NpgsqlTimeout.Check()
   at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
   at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|213_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
   at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Additional context

We would like to understand if this is a typical issue when performing load testing under these conditions, or if we may have missed something in our configuration. Could you kindly provide any advice or recommendations on how we could address this problem and optimize the system for higher traffic?

Thank you for your support, and we look forward to your guidance.