Occasional MultipleActiveResultSets Exceptions On Identity Server Endpoints

[hunterbdoerr]

Which version of Duende IdentityServer are you using? 6.0

Which version of .NET are you using? 6.0

Describe the bug When calling the GET token endpoint, there is an occasional MultipleActiveResultSets exception thrown.

Reading information about the specific error message unrelated to this workflow, the most common culprit is an async method that is not awaited. This would occur in the same transient dbcontext created for the request attempting to execute a request on top of itself.

Log output/exception with stacktrace

Severity level: Information, Message: UnhandledExceptionEvent { Details: "System.InvalidOperationException: The connection does not support MultipleActiveResultSets.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToArrayAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Duende.IdentityServer.EntityFramework.Stores.PersistedGrantStore.StoreAsync(PersistedGrant token) in /_/src/EntityFramework.Storage/Stores/PersistedGrantStore.cs:line 59
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.StoreItemByHashedKeyAsync(String hashedKey, T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Nullable`1 expiration, Nullable`1 consumedTime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 227
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.CreateItemAsync(T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Int32 lifetime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 173
   at Duende.IdentityServer.Stores.DefaultRefreshTokenStore.StoreRefreshTokenAsync(RefreshToken refreshToken) in /_/src/IdentityServer/Stores/Default/DefaultRefreshTokenStore.cs:line 43
   at Duende.IdentityServer.Services.DefaultRefreshTokenService.CreateRefreshTokenAsync(RefreshTokenCreationRequest request) in /_/src/IdentityServer/Services/Default/DefaultRefreshTokenService.cs:line 223
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(ValidatedTokenRequest request) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 454
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.ProcessTokenRequestAsync(TokenRequestValidationResult validationResult) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 336
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.ProcessAsync(TokenRequestValidationResult request) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 98
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 128
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 81
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101", Exception: InvalidOperationException { TargetSite: System.Data.Common.DbDataReader <ExecuteDbDataReaderAsync>b__209_0(System.Threading.Tasks.Task`1[Microsoft.Data.SqlClient.SqlDataReader]), Message: "The connection does not support MultipleActiveResultSets.", Data: [DictionaryEntry { Key: Object {  }, Value: null }], InnerException: null, HelpLink: null, Source: "Microsoft.Data.SqlClient", HResult: -2146233079, StackTrace: "   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToArrayAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Duende.IdentityServer.EntityFramework.Stores.PersistedGrantStore.StoreAsync(PersistedGrant token) in /_/src/EntityFramework.Storage/Stores/PersistedGrantStore.cs:line 59
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.StoreItemByHashedKeyAsync(String hashedKey, T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Nullable`1 expiration, Nullable`1 consumedTime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 227
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.CreateItemAsync(T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Int32 lifetime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 173
   at Duende.IdentityServer.Stores.DefaultRefreshTokenStore.StoreRefreshTokenAsync(RefreshToken refreshToken) in /_/src/IdentityServer/Stores/Default/DefaultRefreshTokenStore.cs:line 43
   at Duende.IdentityServer.Services.DefaultRefreshTokenService.CreateRefreshTokenAsync(RefreshTokenCreationRequest request) in /_/src/IdentityServer/Services/Default/DefaultRefreshTokenService.cs:line 223
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(ValidatedTokenRequest request) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 454
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.ProcessTokenRequestAsync(TokenRequestValidationResult validationResult) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 336
   at Duende.IdentityServer.ResponseHandling.TokenResponseGenerator.ProcessAsync(TokenRequestValidationResult request) in /_/src/IdentityServer/ResponseHandling/Default/TokenResponseGenerator.cs:line 98
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 128
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 81
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101" }, Category: "Error", Name: "Unhandled Exception", EventType: Error, Id: 3000, Message: "The connection does not support MultipleActiveResultSets.", ActivityId: "40000f82-0000-f600-b63f-84710c7967bb", TimeStamp: 05/31/2024 19:07:35, ProcessId: 7736, LocalIpAddress: "xxx", RemoteIpAddress: "x" }

[RolandGuijt]

In your connection string, do you have MultipleActiveResultSets set to true?

Please see an example connection string here.

[hunterbdoerr]

The connection string sets MultipleActiveResultSets=false.

[RolandGuijt]

Is there a reason you can't set it to true? If you do that it'll probably resolve your issue.

[robertmadril]

Jumping in here since @hunterbdoerr and I are contributors on the same project.

We had been hesitant to move forward with that solution given that connection string is shared with other projects. The primary identified source of risk comes from transaction savepoints being disabled when MARS is enabled as documented here for EF functionality..

That being said, enabling MARS does seem to resolve this issue, and we can try to work around the other identified risk areas. However, if there are any other known solutions or other ways of approaching this issue, any additional information would be greatly helpful.