Performance Degradation During Login and Redirect in IdentityServer

[CangPham]

Which version of Duende IdentityServer are you using?

• Version 6.0.2

Which version of .NET are you using?

• Dotnet 6

Describe the bug

Summary: We've encountered significant performance degradation in our IdentityServer instance during the login process, especially when redirecting back to the client. This issue seems to stem from database inefficiencies, particularly when handling clients with a large number of redirect URIs.

Detailed Description: While analyzing the logs and performance during the login process, we observed the following:

Slow Redirects:

The issue occurs when attempting to log in to IdentityServer, resulting in a very long wait time when redirecting back to the client. The delay exceeds 11 seconds when there are more than 100 redirect URIs associated with a single client. Log Analysis:

The log file shows a 4-second delay, even though the query execution is reported to take only 4 ms:

    Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.TryReadColumnInternal(int i, bool readHeaderOnly)   Unknown
    Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(System.Threading.Tasks.Task task, object state)    Unknown
    Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.InvokeAsyncCall<bool>(Microsoft.Data.SqlClient.SqlDataReader.AAsyncCallContext<bool> context)   Unknown
    Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.ReadAsync(System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Storage.RelationalDataReader.ReadAsync(System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable<Duende.IdentityServer.EntityFramework.Entities.Client>.AsyncEnumerator.MoveNextAsync()   Unknown
    [Resuming Async Method] 
    [External Code] 
    Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.CompleteAsyncCall<bool>(System.Threading.Tasks.Task<bool> task, Microsoft.Data.SqlClient.SqlDataReader.AAsyncCallContext<bool> context) Unknown
    [External Code] 
    [Async Call Stack]  
    [Async] Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync<Duende.IdentityServer.EntityFramework.Entities.Client>(System.Linq.IQueryable<Duende.IdentityServer.EntityFramework.Entities.Client> source, System.Threading.CancellationToken cancellationToken)   Unknown
    [Async] Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToArrayAsync<Duende.IdentityServer.EntityFramework.Entities.Client>(System.Linq.IQueryable<Duende.IdentityServer.EntityFramework.Entities.Client> source, System.Threading.CancellationToken cancellationToken)  Unknown
>   [Async] Duende.IdentityServer.EntityFramework.Storage.dll!Duende.IdentityServer.EntityFramework.Stores.ClientStore.FindClientByIdAsync(string clientId) Line 74 C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Stores.ValidatingClientStore<Duende.IdentityServer.EntityFramework.Stores.ClientStore>.FindClientByIdAsync(string clientId) Line 54 C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Stores.IClientStoreExtensions.FindEnabledClientByIdAsync(Duende.IdentityServer.Stores.IClientStore store, string clientId) Line 23  C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Validation.AuthorizeRequestValidator.LoadClientAsync(Duende.IdentityServer.Validation.ValidatedAuthorizeRequest request) Line 218   C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Validation.AuthorizeRequestValidator.ValidateAsync(System.Collections.Specialized.NameValueCollection parameters, System.Security.Claims.ClaimsPrincipal subject) Line 78   C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Endpoints.AuthorizeEndpointBase.ProcessAuthorizeRequestAsync(System.Collections.Specialized.NameValueCollection parameters, System.Security.Claims.ClaimsPrincipal user, bool checkConsentResponse) Line 88 C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(Microsoft.AspNetCore.Http.HttpContext context) Line 50 C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context, Duende.IdentityServer.Hosting.IEndpointRouter router, Duende.IdentityServer.Services.IUserSession userSession, Duende.IdentityServer.Services.IEventService events, Duende.IdentityServer.Services.IIssuerNameService issuerNameService, Duende.IdentityServer.Services.ISessionCoordinationService sessionCoordinationService) Line 98  C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context, Microsoft.AspNetCore.Authentication.IAuthenticationSchemeProvider schemes) Line 94    C#
    [External Code] 
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) Line 47    C#
    [Async] Duende.IdentityServer.dll!Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) Line 27 C#
    [External Code] 

Manually executing the same query returns 50,000 rows and takes 4 seconds, indicating a potential issue with the Identity database.

SELECT [c].[Id], [c].[AbsoluteRefreshTokenLifetime], ... , [c8].[RedirectUri]
FROM [Clients] AS [c]
LEFT JOIN [ClientCorsOrigins] AS [c0] ON [c].[Id] = [c0].[ClientId]
LEFT JOIN [ClientGrantTypes] AS [c1] ON [c].[Id] = [c1].[ClientId]
LEFT JOIN [ClientScopes] AS [c2] ON [c].[Id] = [c2].[ClientId]
LEFT JOIN [ClientClaims] AS [c3] ON [c].[Id] = [c3].[ClientId]
LEFT JOIN [ClientSecrets] AS [c4] ON [c].[Id] = [c4].[ClientId]
LEFT JOIN [ClientIdPRestrictions] AS [c5] ON [c].[Id] = [c5].[ClientId]
LEFT JOIN [ClientPostLogoutRedirectUris] AS [c6] ON [c].[Id] = [c6].[ClientId]
LEFT JOIN [ClientProperties] AS [c7] ON [c].[Id] = [c7].[ClientId]
LEFT JOIN [ClientRedirectUris] AS [c8] ON [c].[Id] = [c8].[ClientId]
WHERE [c].[ClientId] = @__clientId_0
ORDER BY [c].[Id], [c0].[Id], [c1].[Id], [c2].[Id], [c3].[Id], [c4].[Id], [c5].[Id], [c6].[Id], [c7].[Id], [c8].[Id]

Our database contains a large number of different ClientRedirectUris and ClientPostLogoutRedirectUris that share the same clientId. This results in the query returning a significant number of rows, contributing to the observed performance issues.

In particular, the problem becomes severe when the client has more than 100 redirect URIs, causing the login process to take over 11 seconds to complete the redirect.

To Reproduce

Seed Data:

We have a function that seeds data every time a new version is installed. During development, we change the URI of IdentityServer according to the current version for easier tracking (e.g., gfdevsuite2020, gfdevsuite2021, etc.). Development Process:

As a result of these URI changes, multiple entries are created in the ClientRedirectUris and ClientPostLogoutRedirectUris tables with different URIs but the same clientId. Login Attempt:

Attempt to log in to IdentityServer with a client that has more than 100 redirect URIs associated with it. Observe the delay in redirecting back to the client, which can exceed 11 seconds.

Expected behavior

Database Performance:

• The combination of multiple LEFT JOIN operations and the presence of many ClientRedirectUris and ClientPostLogoutRedirectUris associated with the same clientId is likely causing the issue. Redundant data in these tables, possibly due to changes in the server’s PC Name over time, could be exacerbating the problem.

Request for Assistance:

• Could you please investigate the root cause of this performance issue? Additionally, we would appreciate guidance on optimizing these queries.

FYI https://github.com/cakkermans @cakkermans

[RolandGuijt]

Can you please tell us more about the scenario you're in? How does changing IdentityServer's URI affect the ClientRedirectUris/ClientPostLogoutRedirectUris? Having a client with so many redirect uris is not something we recommend also from a security standpoint.

[CangPham]

Hi @RolandGuijt ,

In our scenario, every time the company releases a new version of the software, we clone a virtual machine that installs the software with a name that includes the software name and version for better management.

Step by step:

Version Release: For each new version release, we clone the virtual machine, giving it a unique name like gfdevsuite2020, gfdevsuite2021, etc. Development Mode: In development, each developer works on a machine with a different name, but all machines share the same database. Impact on Redirect URIs: As a result, we end up with many redirect URIs in the ClientRedirectUris and ClientPostLogoutRedirectUris tables for a single client because the client is used across multiple machine instances, each with a different URI.

[RolandGuijt]

The Linq query that retrieves clients has been optimized in 6.0.4. Can you please upgrade to the latest 6.0 version, 6.0.5 and see if the problem persists? You should be able to upgrade without doing any migrations or modifications. We recommend you update to the latest 6 version 6.3.10 as soon as possible as it fixes security-related issues too. But that would require migrations and possibly modifications. Please see our upgrade guide.