search

DuendeSoftware / Support

Support for Duende Software products
21 stars 0 forks source link

Unexpected Timeout #641

Closed LKingstonEdgeCo closed 1 year ago

LKingstonEdgeCo commented 1 year ago

Which version of Duende IdentityServer are you using? Duende.IdentityServer 6.2.3 Which version of .NET are you using? .NET 6 Describe the bug

We are working on upgrading from .NET core 3.1 to .NET 6. At the same time we are upgrading from IdentityServer4 3.0.2 to Duende 6.2.3. Our IS application is separated into 2 parts: our log in web application and our Identity API. We've successfully ran the project locally but when we deploy to our DEV environment, we keep getting a Timeout exception when trying to authenticate any of our clients. This error will occasionally happen locally, but usually just runs slow in a local environment. When running locally, I've noticed that there is a lot of time between when we call FindApiResourcesByNameAsync in our web application to when the breakpoint in the API's controller is hit. Once the breakpoint in our API is hit, the business logic and the Entity Framework Core query in the API is executed immediately.

To Reproduce

Create an instance of our IS web application and our Identity API, then attempt to authenticate the Identity API.

Expected behavior

A quick authentication (no more than 1 second) of the API.

Log output/exception with stacktrace

Category: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware
EventId: 1
SpanId: 7e859eb89679e406
TraceId: 9b50187cbb49774db8d6fc4f5839f4ed
ParentId: 9b564ed5fc8d013c
RequestId: 80000b96-0001-fa00-b63f-84710c7967bb
RequestPath: /identity/connect/introspect

An unhandled exception has occurred while executing the request.

Exception: 
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at UnifiedTrust.Common.Data.API.Helpers.APIHelper.GetAsync[ObjectType](String url, CancellationToken cancellationToken, Object[] parameters)
   at RS.Identity.Web.Services.ResourceStore.FindApiResourcesByNameAsync(IEnumerable`1 apiResourceNames) in D:\tfsbuildagent1\_work\514\s\src\RS.Identity.Web\Services\ResourceStore.cs:line 125
   at Duende.IdentityServer.Stores.CachingResourceStore`1.<FindApiResourcesByNameAsync>b__13_0(IEnumerable`1 names) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 174
   at Duende.IdentityServer.Stores.CachingResourceStore`1.<>c__DisplayClass16_0`1.<<FindItemsAsync>b__0>d.MoveNext() in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 240
--- End of stack trace from previous location ---
   at Duende.IdentityServer.Services.DefaultCache`1.GetOrAddAsync(String key, TimeSpan duration, Func`1 get) in /_/src/IdentityServer/Services/Default/DefaultCache.cs:line 149
   at Duende.IdentityServer.Stores.CachingResourceStore`1.FindItemsAsync[TItem](IEnumerable`1 names, ICache`1 cache, Func`2 getResourcesFunc, Func`2 getFromResourcesFunc, Func`2 getNameFunc, String allCachePrefix) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 240
   at Duende.IdentityServer.Stores.CachingResourceStore`1.FindApiResourcesByNameAsync(IEnumerable`1 apiResourceNames) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 173
   at Duende.IdentityServer.Validation.ApiSecretValidator.ValidateAsync(HttpContext context) in /_/src/IdentityServer/Validation/Default/ApiSecretValidator.cs:line 69
   at Duende.IdentityServer.Endpoints.IntrospectionEndpoint.ProcessIntrospectionRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/IntrospectionEndpoint.cs:line 94
   at Duende.IdentityServer.Endpoints.IntrospectionEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/IntrospectionEndpoint.cs:line 80
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 121
   at Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) in /_/src/IdentityServer/Hosting/MutualTlsEndpointMiddleware.cs:line 95
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/DynamicProviders/DynamicSchemes/DynamicSchemeAuthenticationMiddleware.cs:line 48
   at Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/BaseUrlMiddleware.cs:line 28
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at RS.Identity.Web.Middleware.RedirectValidationMiddleware.Invoke(HttpContext context, IWhiteLabelPartnerStore whiteLabelPartnerStore, IOptions`1 cacheOptions, ILogger`1 logger) in D:\tfsbuildagent1\_work\514\s\src\RS.Identity.Web\Middleware\RedirectValidationMiddleware.cs:line 52
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Additional context

Here is the FindApiResourcesByNameAsync method in our ResourceStore of our IS web application:

public async Task<IEnumerable<ApiResource>> FindApiResourcesByNameAsync(IEnumerable<string> apiResourceNames)
        {
            if (apiResourceNames == null)
            {
                throw new ArgumentNullException(nameof(apiResourceNames));
            }

            var result = await _apiHelper.GetAsync<List<ApiResource>>("api/resources/FindApiResourcesByNameAsync?apiResourceNames={0}", string.Join(",", apiResourceNames));
            return result ?? Enumerable.Empty<ApiResource>();
        }

The ResourceController in the API:

public async Task<ActionResult<ApiResource>> FindApiResourcesByNameAsync([FromQuery] string apiResourceNames)
        {
            if (string.IsNullOrWhiteSpace(apiResourceNames))
                return BadRequest();

            var list = apiResourceNames.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries).ToList();
            var resource = await _resourceStore.FindApiResourcesByNameAsync(list);

            if (resource == null)
                return NotFound();

            return Ok(resource);
        }

And the _resourceStore.FindApiResourcesByNameAsync() method being called in the controller:

public async Task<IEnumerable<ApiResource>> FindApiResourcesByNameAsync(IEnumerable<string> apiResourceNames)
        {
            if (apiResourceNames == null) throw new ArgumentNullException(nameof(apiResourceNames));

            return _mapper.Map<IEnumerable<ApiResource>>((await _dbContext.ApiResources.AsNoTracking().ToListAsync())?.Where(x => apiResourceNames.Contains(x.Resource.Name)).Select(x => x.Resource));

        }
josephdecock commented 1 year ago

This sounds like a caching problem with your configuration store. Since configuration data typically changes infrequently, caching it is usually appropriate. We describe how to do so in our documentation here.

LKingstonEdgeCo commented 1 year ago

That was my first thought too, but I'm not sure what needs to change about our caching configuration. Here is the section of our startup file that has those settings:

// configure identity server with in-memory stores, keys, clients and scopes
            var idSrvBuilder = services.AddIdentityServer(options =>
                {
                    options.LicenseKey = Configuration.GetValue<string>("DuendeLicenseKey");
                    options.UserInteraction.LoginUrl = PageRoutes.LoginPages.Login;
                    options.UserInteraction.LogoutUrl = PageRoutes.Logout;
                    options.UserInteraction.ErrorUrl = PageRoutes.ErrorPage;
                    if (!HostingEnvironment.IsProduction())
                    {
                        //options.Caching.ClientStoreExpiration = TimeSpan.FromMinutes(1);
                        //options.Caching.ResourceStoreExpiration = TimeSpan.FromMinutes(1);
                    }
                    options.Events.RaiseErrorEvents = true;
                    options.Events.RaiseFailureEvents = true;
                    options.Events.RaiseSuccessEvents = true;
                    // Below settings allow us to authenticate calls to Identity API from this site without callback deadlocks
                    // to the discovery document
                    options.Discovery.ShowIdentityScopes = false;
                    options.Discovery.ShowApiScopes = false;
                    options.Discovery.ShowClaims = false;

                    // support for older clients
                    //options.EmitLegacyResourceAudienceClaim = true;
                    options.EmitStaticAudienceClaim = true;

                })
                .AddClientStore<ClientStore>()
                .AddResourceStore<ResourceStore>()
                .AddInMemoryCaching()
                .AddClientStoreCache<ClientStore>()
                .AddResourceStoreCache<ResourceStore>()
                .AddRedirectUriValidator<StrictRedirectUriValidatorAppAuth>()
                //.AddAspNetIdentity<User>()    // NB 02/25/2023 Is this really needed !??
                .AddServiceProviderStore<ServiceProviderStore>();
LKingstonEdgeCo commented 1 year ago

@josephdecock I've removed caching from my startup, but am still getting a timeout error. Currently, my startup looks like this:

.AddClientStore<ClientStore>()
.AddResourceStore<ResourceStore>()
//.AddInMemoryCaching()
//.AddClientStoreCache<ClientStore>()
//.AddResourceStoreCache<ResourceStore>()

And here is the exception now:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at UnifiedTrust.Common.Data.API.Helpers.APIHelper.GetAsync[ObjectType](String url, CancellationToken cancellationToken, Object[] parameters)
   at RS.Identity.Web.Services.ClientStore.FindClientByIdAsync(String clientId) in C:\Code\Git\WebApps-External\RS.Identity.Web\src\RS.Identity.Web\Services\ClientStore.cs:line 48
   at Duende.IdentityServer.Stores.ValidatingClientStore`1.FindClientByIdAsync(String clientId) in /_/src/IdentityServer/Stores/ValidatingClientStore.cs:line 54
   at Duende.IdentityServer.Stores.IClientStoreExtensions.FindEnabledClientByIdAsync(IClientStore store, String clientId) in /_/src/IdentityServer/Extensions/IClientStoreExtensions.cs:line 23
   at Duende.IdentityServer.Validation.ClientSecretValidator.ValidateAsync(HttpContext context) in /_/src/IdentityServer/Validation/Default/ClientSecretValidator.cs:line 72
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 89
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 75
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 117
   at Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) in /_/src/IdentityServer/Hosting/MutualTlsEndpointMiddleware.cs:line 94
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/DynamicProviders/DynamicSchemes/DynamicSchemeAuthenticationMiddleware.cs:line 47
   at Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/BaseUrlMiddleware.cs:line 27
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
   at RS.Identity.Web.Middleware.RedirectValidationMiddleware.Invoke(HttpContext context, IWhiteLabelPartnerStore whiteLabelPartnerStore, IOptions`1 cacheOptions, ILogger`1 logger) in C:\Code\Git\WebApps-External\RS.Identity.Web\src\RS.Identity.Web\Middleware\RedirectValidationMiddleware.cs:line 52
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

My API is being hit, but not before the socket times out. Any help would be appreciated!

josephdecock commented 1 year ago

This stack trace looks like your store implementation is timing out calling an api. I would start by verifying that the api you're calling is responsive, outside of IdentityServer. You could also try switching to the in memory client store as a test - if the issue goes away with the in memory store, that would confirm that the problem is in the client store.

LKingstonEdgeCo commented 1 year ago

The API is responsive and returns results. Right now we are able to log in with a user through identity server, but can only sometimes authenticate with our API. We are seeing a few errors being returned from the ../connect/introspect endpoint.

error 1:

Category: Duende.IdentityServer.ResponseHandling.IntrospectionResponseGenerator
EventId: 0
SpanId: d39d4b32b510d6b9
TraceId: 092f995b5d1ec0376f6db370a07d073b
ParentId: aa0e12fc553ca0c2
RequestId: 8000010d-0000-d900-b63f-84710c7967bb
RequestPath: /identity6/connect/introspect

error 2:

Expected scope {"Required":false,"Emphasize":false,"Enabled":true,"Name":"api.identity.fullaccess","DisplayName":"Identity API - Full Access","Description":"Identity API - Full Access","ShowInDiscoveryDocument":true,"UserClaims":[],"Properties":{}}, {"Required":false,"Emphasize":false,"Enabled":true,"Name":"api.identity.readwrite","DisplayName":"Identity API - Read/Write","Description":"Identity API - Read/Write","ShowInDiscoveryDocument":true,"UserClaims":[],"Properties":{}} is missing in token

error 3:

Token Introspection Failure (2021), Details: Expected scopes are missing

Our API is still logging the HttpClient.Timeout error when this happens. Any idea what could be causing this error?

josephdecock commented 1 year ago

These errors all indicate that the API is trying to introspect a token that it shouldn't. Each API resource defines the scopes it supports using the Scopes property. The semantic here is that the scopes in that collection are scopes that can give some access to the API. When a token is introspected, the scopes within the token are compared to the scopes declare on the API resource, and if there is no overlap, then the introspection fails.

To fix this problem, either add scopes that this API wants to use to the API Resource definition, or change the client application so that it requests a token containing scopes that this API supports.

josephdecock commented 1 year ago

Is anything further needed on this issue, or should we close it?

josephdecock commented 1 year ago

Closing, but feel free to reopen if necessary.

josephdecock commented 1 year ago

@LKingstonEdgeCo - I've reopened this issue based on your email. Since it's been a while since my last post, can you update me on what you need/how I can help?

LKingstonEdgeCo commented 1 year ago

@josephdecock - Sorry to leave you hanging. I'm not sure our issue is being caused by IS, but here is the latest error log: 

fail: UnifiedTrust.Identity.Services.Logging.IdServerEventSink[0]
      => SpanId:1b50ea5541b33fda, TraceId:e8f87dc7a5b1bcc35b02f0af281f36b1, ParentId:0000000000000000 => ConnectionId:0HMT2A0S7FLN7 => RequestPath:/connect/token RequestId:0HMT2A0S7FLN7:00000007
      Unhandled Exception (3000), Details: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
crit: Duende.IdentityServer.Hosting.IdentityServerMiddleware[0]
      => SpanId:1b50ea5541b33fda, TraceId:e8f87dc7a5b1bcc35b02f0af281f36b1, ParentId:0000000000000000 => ConnectionId:0HMT2A0S7FLN7 => RequestPath:/connect/token RequestId:0HMT2A0S7FLN7:00000007
      Unhandled exception: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
      System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
       ---> System.TimeoutException: The operation was canceled.
       ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
       ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
         at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
         at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
         at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at RS.Common.Data.API6.Helpers.APIHelper.GetAsync[ObjectType](String url, CancellationToken cancellationToken, Object[] parameters) in C:\Repos\RS.Common.Data.API6\src\RS.Common.Data.API6\Helpers\APIHelper.cs:line 506
         at RS.Identity.Web.Services.ClientStore.FindClientByIdAsync(String clientId) in C:\Repos\RS.Identity.Web\src\RS.Identity.Web\Services\ClientStore.cs:line 50
         at Duende.IdentityServer.Stores.ValidatingClientStore`1.FindClientByIdAsync(String clientId) in /_/src/IdentityServer/Stores/ValidatingClientStore.cs:line 54
         at Duende.IdentityServer.Stores.CachingClientStore`1.<>c__DisplayClass4_0.<<FindClientByIdAsync>b__0>d.MoveNext() in /_/src/IdentityServer/Stores/Caching/CachingClientStore.cs:line 51
      --- End of stack trace from previous location ---
         at Duende.IdentityServer.Services.DefaultCache`1.GetOrAddAsync(String key, TimeSpan duration, Func`1 get)
         at Duende.IdentityServer.Stores.CachingClientStore`1.FindClientByIdAsync(String clientId) in /_/src/IdentityServer/Stores/Caching/CachingClientStore.cs:line 49
         at Duende.IdentityServer.Stores.IClientStoreExtensions.FindEnabledClientByIdAsync(IClientStore store, String clientId) in /_/src/IdentityServer/Extensions/IClientStoreExtensions.cs:line 23
         at Duende.IdentityServer.Validation.ClientSecretValidator.ValidateAsync(HttpContext context) in /_/src/IdentityServer/Validation/Default/ClientSecretValidator.cs:line 72
         at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 89
         at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 75
         at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      => SpanId:1b50ea5541b33fda, TraceId:e8f87dc7a5b1bcc35b02f0af281f36b1, ParentId:0000000000000000 => ConnectionId:0HMT2A0S7FLN7 => RequestPath:/connect/token RequestId:0HMT2A0S7FLN7:00000007
      An unhandled exception has occurred while executing the request.
      System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
       ---> System.TimeoutException: The operation was canceled.
       ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
       ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
         at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
         at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
         at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at RS.Common.Data.API6.Helpers.APIHelper.GetAsync[ObjectType](String url, CancellationToken cancellationToken, Object[] parameters) in C:\Repos\RS.Common.Data.API6\src\RS.Common.Data.API6\Helpers\APIHelper.cs:line 506
         at RS.Identity.Web.Services.ClientStore.FindClientByIdAsync(String clientId) in C:\Repos\RS.Identity.Web\src\RS.Identity.Web\Services\ClientStore.cs:line 50
         at Duende.IdentityServer.Stores.ValidatingClientStore`1.FindClientByIdAsync(String clientId) in /_/src/IdentityServer/Stores/ValidatingClientStore.cs:line 54
         at Duende.IdentityServer.Stores.CachingClientStore`1.<>c__DisplayClass4_0.<<FindClientByIdAsync>b__0>d.MoveNext() in /_/src/IdentityServer/Stores/Caching/CachingClientStore.cs:line 51
      --- End of stack trace from previous location ---
         at Duende.IdentityServer.Services.DefaultCache`1.GetOrAddAsync(String key, TimeSpan duration, Func`1 get)
         at Duende.IdentityServer.Stores.CachingClientStore`1.FindClientByIdAsync(String clientId) in /_/src/IdentityServer/Stores/Caching/CachingClientStore.cs:line 49
         at Duende.IdentityServer.Stores.IClientStoreExtensions.FindEnabledClientByIdAsync(IClientStore store, String clientId) in /_/src/IdentityServer/Extensions/IClientStoreExtensions.cs:line 23
         at Duende.IdentityServer.Validation.ClientSecretValidator.ValidateAsync(HttpContext context) in /_/src/IdentityServer/Validation/Default/ClientSecretValidator.cs:line 72
         at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 89
         at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 75
         at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
         at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 117
         at Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) in /_/src/IdentityServer/Hosting/MutualTlsEndpointMiddleware.cs:line 94
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/DynamicProviders/DynamicSchemes/DynamicSchemeAuthenticationMiddleware.cs:line 47
         at Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/BaseUrlMiddleware.cs:line 27
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at RS.Identity.Web.Middleware.RedirectValidationMiddleware.Invoke(HttpContext context, IWhiteLabelPartnerStore whiteLabelPartnerStore, IOptions`1 cacheOptions, ILogger`1 logger) in C:\Repos\RS.Identity.Web\src\RS.Identity.Web\Middleware\RedirectValidationMiddleware.cs:line 52
         at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      => SpanId:31856b93a2aa614c, TraceId:e8f87dc7a5b1bcc35b02f0af281f36b1, ParentId:471fc07b6ed9d126 => ConnectionId:0HMT2A0S7FLN9 => RequestPath:/connect/introspect RequestId:0HMT2A0S7FLN9:00000002
      An unhandled exception has occurred while executing the request.
      System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
       ---> System.TimeoutException: The operation was canceled.
       ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
       ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
         at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
         at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
         at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at RS.Common.Data.API6.Helpers.APIHelper.GetAsync[ObjectType](String url, CancellationToken cancellationToken, Object[] parameters) in C:\Repos\RS.Common.Data.API6\src\RS.Common.Data.API6\Helpers\APIHelper.cs:line 506
         at RS.Identity.Web.Services.ResourceStore.FindApiResourcesByNameAsync(IEnumerable`1 apiResourceNames) in C:\Repos\RS.Identity.Web\src\RS.Identity.Web\Services\ResourceStore.cs:line 125
         at Duende.IdentityServer.Stores.CachingResourceStore`1.<FindApiResourcesByNameAsync>b__13_0(IEnumerable`1 names) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 174
         at Duende.IdentityServer.Stores.CachingResourceStore`1.<>c__DisplayClass16_0`1.<<FindItemsAsync>b__0>d.MoveNext() in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 240
      --- End of stack trace from previous location ---
         at Duende.IdentityServer.Services.DefaultCache`1.GetOrAddAsync(String key, TimeSpan duration, Func`1 get)
         at Duende.IdentityServer.Stores.CachingResourceStore`1.FindItemsAsync[TItem](IEnumerable`1 names, ICache`1 cache, Func`2 getResourcesFunc, Func`2 getFromResourcesFunc, Func`2 getNameFunc, String allCachePrefix) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 240
         at Duende.IdentityServer.Stores.CachingResourceStore`1.FindApiResourcesByNameAsync(IEnumerable`1 apiResourceNames) in /_/src/IdentityServer/Stores/Caching/CachingResourceStore.cs:line 173
         at Duende.IdentityServer.Validation.ApiSecretValidator.ValidateAsync(HttpContext context) in /_/src/IdentityServer/Validation/Default/ApiSecretValidator.cs:line 69
         at Duende.IdentityServer.Endpoints.IntrospectionEndpoint.ProcessIntrospectionRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/IntrospectionEndpoint.cs:line 94
         at Duende.IdentityServer.Endpoints.IntrospectionEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/IntrospectionEndpoint.cs:line 80
         at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
         at Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) in /_/src/IdentityServer/Hosting/MutualTlsEndpointMiddleware.cs:line 94
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/DynamicProviders/DynamicSchemes/DynamicSchemeAuthenticationMiddleware.cs:line 47
         at Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/BaseUrlMiddleware.cs:line 27
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
         at RS.Identity.Web.Middleware.RedirectValidationMiddleware.Invoke(HttpContext context, IWhiteLabelPartnerStore whiteLabelPartnerStore, IOptions`1 cacheOptions, ILogger`1 logger) in C:\Repos\RS.Identity.Web\src\RS.Identity.Web\Middleware\RedirectValidationMiddleware.cs:line 52
         at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

The RS.Common.Data.API6.Helpers.APIHelper.GetAsync call in the stack trace is our main method used to call all of our various APIs. It's working correctly all over this application except for when we call our implementation of IClientStore.FindClientByIdAsync() or IResourceStore.FindApiResourcesByNameAsync(). Here are those implementations:

        public async Task<Client> FindClientByIdAsync(string clientId)
        {
            return await _apiHelper.GetAsync<Client>("api/clients/findByClientId?clientId={0}", clientId);
        }
        public async Task<IEnumerable<ApiResource>> FindApiResourcesByNameAsync(IEnumerable<string> apiResourceNames)
        {
            if (apiResourceNames == null)
            {
                throw new ArgumentNullException(nameof(apiResourceNames));
            }

            var result = await _apiHelper.GetAsync<List<ApiResource>>("api/resources/FindApiResourcesByNameAsync?apiResourceNames={0}", string.Join(",", apiResourceNames));
            return result ?? Enumerable.Empty<ApiResource>();
        }

Any help would be greatly appreciated!

josephdecock commented 1 year ago

This looks to be an issue in your API or the api helper. Assuming that your api helper is making http calls to web apis, the questions that I would investigate next are things like this:

  1. Is the api helper actually making an http request in this case?
  2. Is the http request being routed correctly to your api/is there some sort of networking issue (firewalls, etc) in the middle that is keeping the request from reaching the api?
  3. If the request reaches the api successfully, what code is running in the api? What could cause a performance bottleneck there? Does that api log anything?

(No need to answer all these questions here, this is just "for your consideration" as you debug your service).

The other thing that you might do to reduce the load on the api is add caching to the identity server stores. This could significantly reduce the number of requests to the API in the first place.

josephdecock commented 1 year ago

Any progress on this issue? Did caching help?

LKingstonEdgeCo commented 1 year ago

I believe we've found the root of the problem. As I mentioned above, our Identity Server exists in a web application with an API on the back end that handles database calls. It seems like our timeout exception is happening when the API tries to authorize calls from the IS web application, because the two applications get stuck in a loop of the web application calling the API, then the API calling the web application to authorize the call, which causes the web application to call the API, and so on.

Here is the exception we are receiving: image

This seems to be related to this code change we made to the API's startup file, specifically the change from using AddIdentityServerAuthentication() to AddOAuth2Introspection(): image

Here is the IS configuration from the web application as well: image

The question I have now is: how do we secure the API that our IS web application relies on for its configuration and operational data?

josephdecock commented 1 year ago

Have you considered moving the database access out of the API and into IdentityServer itself? The configuration and operational stores really are IdentityServer specific data, so in most cases the only thing that should be accessing those stores is IdentityServer. So a question for you to consider is, what are you gaining by adding an API in between?

If you decide that you really want to keep that API, it will have to authenticate requests from IdentityServer without using tokens issued by IdentityServer itself. You could have some kind of shared secret that you use to add a signature to your http requests in an http header perhaps. That's beyond the scope of what IdentityServer does though. The simplest thing probably is to consolidate the data store inside IdentityServer.

josephdecock commented 1 year ago

Any update here?

AndersAbel commented 1 year ago

Closing due to inactivity. Please feel free to reopen if needed.