IServerUrls-related issue after IdentityServer4 -> Duende IdentityServer v6, v7 migration

[promim]

Which version of Duende IdentityServer are you using? 7.0.4

Which version of .NET are you using? 8.0.3

Describe the bug We've just migrated our IdentityServer4 implementation to Duende IdentityServer v6 and v7 subsequently. We have had a development setup, where we run the IdentityServer implementation in a Docker container and an MVC client application (with configured OIDC) in another Docker container. It appears that after the required IServerUrls change in one of the Duende IdentityServer v6 subversions, this setup no longer works as before.

To Reproduce

Steps to reproduce the behavior.

The IdentityServer implementation (referred to as oauth) is available at https://localhost:6501 (which is also the IServerUrls.Origin implementation) The MVC client application (referred to as devframe) is available at https://localhost:31111

docker-compose looks like this

devframe:
  container_name: xx-oauth.devframe
  hostname: devframe
  image: temp.devframe
  build: .
  pull_policy: always
  restart: unless-stopped
  environment:
    - TZ="Europe/Copenhagen"
    - ASPNETCORE_ENVIRONMENT="Development"
    - ASPNETCORE_Kestrel__Certificates__Default__Password="password"
    - ASPNETCORE_Kestrel__Certificates__Default__Path="/https/cert-aspnetcore.pfx"
    - ASPNETCORE_URLS="https://+;http://+"
  ports:
    - "31111:443"
    - "31110:80"
  volumes:
    - "c:/local-cert:/https"

oauth:
  container_name: xx-oauth.oauth
  hostname: oauth
  build: .
  image: temp.oauth
  pull_policy: always
  restart: unless-stopped
  environment:
    - TZ="Europe/Copenhagen"
    - ASPNETCORE_ENVIRONMENT="Development"
    - ASPNETCORE_Kestrel__Certificates__Default__Password="password"
    - ASPNETCORE_Kestrel__Certificates__Default__Path="/https/cert-aspnetcore.pfx"
    - ASPNETCORE_URLS="https://+;http://+"
    - OAUTH_AUTHORITY="https://oauth"
  ports:
    - "6500:80"
    - "6501:443"
  volumes:
    - C:\local-cert:/https

After accessing devframe at https://localhost:31111 in the browser, we trigger an action that challenges the OIDC authentication scheme. Since the authority in the OIDC configuration is set to OAUTH_AUTHORITY (https://oauth), we are able to hit the discovery document endpoint, but since the JWKS URI is set to localhost:6501, devframe from within its container is not able to access it.

A clear and concise description of what you expected to happen.

Something around the time when the IServerUrls change was introduced seems to have caused this problem.

Log output/exception with stacktrace

SocketException: Connection refused
System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)

HttpRequestException: Connection refused (localhost:6501)
System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(string host, int port, HttpRequestMessage initialRequest, bool async, CancellationToken cancellationToken)

IOException: IDX20804: Unable to retrieve document from: 'https://localhost:6501/.well-known/openid-configuration/jwks'.
Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(string address, CancellationToken cancel)

InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://oauth/.well-known/openid-configuration'. Will retry at '23/05/2024 12:40:58 +00:00'. Exception: 'System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:6501/.well-known/openid-configuration/jwks'.
---> System.Net.Http.HttpRequestException: Connection refused (localhost:6501)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---

Additional context

Notice that the communication between the containers seems fine, I can ping any endpoint on oauth from within the devframe container terminal:

Not sure if relevant, but when running the server and the client, I do get the following logs on both devframe and oauth:

Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'https://+;http://+'.
Overriding address(es) 'https://+, http://+'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
Now listening on: http://0.0.0.0:80
Now listening on: https://0.0.0.0:443

Both devframe and oauth are configured to start up with the following setup:

public static void Main(string[] args) => CreateWebHostBuilder(args).Build().Run();

public static IHostBuilder CreateWebHostBuilder(string[] args)
    => Host
        .CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(builder =>
        {
            builder
                .ConfigureAppConfiguration((context, config) =>
                {
                    config
                        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                        .AddEnvironmentVariables();

                    if (args is not null)
                    {
                        config.AddCommandLine(args);
                    }
                })
                .ConfigureWebHostDefaults()
                .UseStartup<Startup>();
        });

public static IWebHostBuilder ConfigureWebHostDefaults(this IWebHostBuilder builder)
    => builder
        .ConfigureKestrel((context, options) => options.AddServerHeader = false)
        .UseKestrel((context, options) =>
        {
            // retrieve certificate from specified path
            var certificate = CertificateHelper.GetX509CertificateFromDefaultConfiguration(context.Configuration);

            if (certificate is not null)
            {
                options.Listen(
                    IPAddress.Any,
                    80,
                    c =>
                    {
                        c.Protocols = HttpProtocols.Http1;
                    });

                options.Listen(
                    IPAddress.Any,
                    443,
                    c =>
                    {
                        c.UseHttps(certificate);
                        c.Protocols = HttpProtocols.Http1AndHttp2;
                    });
            }
            else
            {
                Console.WriteLine("Cannot fetch certificate from keyvault. Kestrel will default to default certificate.");
            }
        });

[RolandGuijt]

For container deployments, the default ASP.NET Core port changed from 80 to 8080. It looks like this is the cause of your problem. It might require configuration updates to either continue using port 80 or to migrate to using 8080.

[promim]

@RolandGuijt I've downgraded both apps (the IdentityServer implementation oauth to .NET 6 and IdentityServer 6.3; the MVC/OIDC client devframe to .NET 6) and the issue is still there. The port breaking change was introduced in .NET 8.

EDIT: Added additional info about how devframe and oauth are configured to start up if that helps explain the setup.

[AndersAbel]

I've reviewed the details you shared once more and this is indeed not caused by the Asp.Net Core default port changes. Those are a very common issue for docker based deployments so that is our first check for any issues when hosting .NET 8 on Docker.

It looks like you have a DNS/host name oauth setup for your IdentityServer. But when IdentityServer processes the request it does not see/accept that host name. I would suggest that you focus on getting the discovery document right. If you access it on https://oauth/.well-known/openid-configuration all the URLs in the discovery document should be to https://oauth/... too. Does your Docker setup include a virtual reverse proxy? Is that configured correctly to emit forwarded headers? Is your IdentityServer host configured correctly to accept forwarded headers? Please see our docs on deployment with proxies for more information on that setup.

If everything is working correctly, IdentityServer should emot URLs (including the JWKs URL) that are under the same host name as the request, e.g. `https://oauth/.well-known/openid-configuration/jwks".

[promim]

@AndersAbel Thanks for the reply.

We do not have a virtual reverse proxy, and never assumed it would be necessary. Especially since the setup was working fine on IdentityServer4.

I was just able to override the endpoints that I receive from the discovery document with the following setup on the MVC client (dev-frame).

builder
                .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options =>
                {
                    options.Configuration = new OpenIdConnectConfiguration
                    {
                        JwksUri = "https://oauth/.well-known/openid-configuration/jwks",
                        TokenEndpoint = "https://oauth/connect/token"
                    };

// ...

It looks like this is only necessary for the endpoints that are called via back-channel (so far I only figured it's the JWKs and the Token endpoints). Is there a better way to do this? Or perhaps a list of endpoints that are called via back-channel by Microsoft's OIDC middleware?

[AndersAbel]

That code would fix the immediate problem, but still doesn't address the root cause, that IdentityServer does not recognize the right host name on the requests. I would recommend getting the host name resolution through IServerUrls correct as that fixes the issue for real. Changing the configuration on the client side is of more a work-around.

[promim]

@AndersAbel That causes other issues that got me worried about other stuff breaking. For example, IdentityServer redirecting to a custom login URL that was set up in the IdentityServer configuration:

services
                .AddIdentityServer(options =>
                {
                    options.UserInteraction.LoginUrl = "/custom-login-url";

// ...

would end up with https://oauth/custom-login-url in the browser, which would never be resolved.

[AndersAbel]

Oh, it looks like this is getting more complicated now. So the oauth address is really not the address that users' browsers will see/show? Having IdentityServer listen/work on different addresses internally and externally is usually a bad idea. The base address is used as the issuer in tokens to identify the OAuth2/OIDC Authority. If different clients/browsers/APIs see different values for that they might reject tokens.

I would recommend making sure that all clients/APIs and users access the IdentityServer using the same host name. If you want docker internal traffic to be possible, I would recommend using split DNS or hosts files to override the public IP address resolution.

[promim]

So the oauth address is really not the address that users' browsers will see/show?

That won't be possible, as "https://oauth" is only supposed to be called by a containerized app such as the MVC client here or an API. As far as IdentityServer is concerned, it receives requests already translated to "localhost:6501". In the browser, the IdentityServer is always called with "localhost:6501".

I still don't understand why this setup was working in IdentityServer4 and not as soon as IServerUrls was introduced. The MVC client always saw the JWKs and the Token endpoints as they were presented in the discovery document (with a localhost:6501 host).

Would it be possible for you to present an example with IdentityServer, an MVC client and an API, each running in a separate Docker container?

[AndersAbel]

This must be a development environment then? Because I guess you do not have IdentityServer actually answering on localhost in production?

When running containerized I've found that it is hard to keep using localhost for interaction with the local environment. A DNS/host name alias is often needed. I think that if you setup a specific host name that can be used both by the users' browsers and the API/client within the containers it should start working.

[promim]

Yes, this is a development environment setup. It would still help a lot if you provided a sample with the mentioned setup. I think it would help a lot of developers struggling with such containerized development configuration.

[AndersAbel]

Our default development setup is to run everything on localhost directly, without containers. It's perfectly possible to set up IdentityServer in a containerized deployment. There are many different ways to do that and there is no solution that fits everyone. Any implementation of IdentityServer needs to make decisions on how to deploy the solution. If opting for containers, virtual networks or anything similar it requires considering the consequences and setting up a proper name resolution. For example, most deployments I've seen in the past uses a reverse proxy and you are not. Either way is fine, I just want to point out that all deployments are different. It would unfortunately not be possible for us to setup and maintain containerized deployment examples that cover the relevant scenarios.

The only generic advice I can give is that the DNS name of the IdentityServer is a fundamental part of the security model. All clients, APIs and user devices should use the same DNS name to communicate with IdentityServer.

[promim]

Thanks for the assistance nevertheless. We decided to go with this for now for the development setup where we disable issuer validation as well, until we figure out a way to use some sort of a reverse proxy to align the use of the same DNS name that's not localhost.

builder
                .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options =>
                {
                    options.Configuration = new OpenIdConnectConfiguration
                    {
                        JwksUri = "https://oauth/.well-known/openid-configuration/jwks",
                        TokenEndpoint = "https://oauth/connect/token"
                    };

// ...

We can close this.