Identity Server BFF on K8S with SSL Termination at Ingress

[dclayton77]

Which version of Duende BFF are you using? 2.0.0

Which version of .NET are you using? .NET 7

Describe the bug

This is not so much a bug but a request for some advice around deployment options. My environment is a K8S cluster where SSL traffic is terminated at the ingress and the HTTP traffic communicated down to the running service over port 80. I have several .NET services that run just fine in this setup.

I am using Authentication Server along with a BFF proxy that provides a React application. The auth server and BFF are both public facing and have valid SSL certificates on the domains. The downstream services are not public facing and BFF can access them within the cluster.

The issue I have is trying to run the BFF framework in this manor. It seems that it wants to run over SSL and will not accommodate my scenario. Using SSL directly into the running service is not really an option due to other factors within the cluster.

Has anyone experienced and solved this scenario. It feels like anyone trying to deploy the BFF pattern to K8S will face the same issue. If anyone could offer up any advice I would very much appreciate it.

The most common error I face is as follows. I get the same if trying to run a mirrored setup using docker-compose locally:

The cookie '.AspNetCore.OpenIdConnect.Nonce.xxxxx' has set 'SameSite=None' and must also set 'Secure'

[AndersAbel]

It is normal deployment practice to use plain http internally within a cluster environment and have the TLS connection terminate at the edge. When doing that, the reverse proxy handling the TLS termination normally sets special headers that indicates to the Asp.Net Core host that the connection is indeed https for the client.

There is a middleware that can be used in Asp.Net Core that inspects these headers and sets HttpContext.Request properties to the values that the browser sees, please see https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-7.0 for more information.

A simple way to validate that this is working on the IdentityServer is to open the discovery document at /.well-known/openid-configuration from a browser. If the forwarded headers middleware isn't correctly setup, the addresses shown in the discovery document are wrong (e.g. with http instead of https).

[dclayton77]

Thanks for your reply, this seems to be exactly the kind of advice I was hoping for. I will investigate this tomorrow and come back with the results 👍

[dclayton77]

I have verified that the IdentityServer is working and can see that all addresses in the discovery document are set to HTTPS. The problem is happening when the login link is clicked. The link navigates to ~/bff/login where I can see that the redirect_uri is passed with the correct address but with HTTP and not HTTPS. I can then see that the same value is passed to the ~/connect/authorise endpoint for IdentityServer.

What I cannot find is where the BFF backend service is setting this value. All of my configuration and environment variables have the HTTPS address so my assumption is that there is something on the BFF side that is changing the protocol. Do you have any ideas as to where this would be happening?

[AndersAbel]

The redirect address is created automatically by the openid connect handler based on the values on HttpContext.Request. If you get http, it indicates forwarded headers are not properly configured on the BFF host.

Do you have UseForwardedHeaders in the BFF pipeline? It should be early, probably first, in the pipeline.

To troubleshoot, you can make a temp API endpoint which returns the the protocol scheme, host and path from HttpContext.Request to get the same diagnostics as using the discovery document on IdentityServer.

[AndersAbel]

@dclayton77 Do you have any update on the status of this?

[dclayton77]

Hi Anders, sorry for the delay in coming back, I've not been able to look at it again until today.

I setup a test endpoint on the IdentityServer and tried two different tests. The first was just to call the endpoint directly from the React application which worked as expected and indicated that HTTPS was being used. The second was to call the endpoint through the BFF service. The second test also worked as expected as you can see from the results output below.

However, if I try to navigate to bff/login, the URL is still not being passed as https. The error in the IdentityServer indicates an invalid redirect uri and the bff service has the following error:

The cookie '.AspNetCore.Correlation.....' has set 'SameSite=None' and must also set 'Secure'

Any other suggestions as to how I can get the redirect uri to be generated correctly?

[dclayton77]

I have managed to get around the issue by adding the following event into the open Id connection configuration. The URL is loaded from config.

[dclayton77]

This still does not fully resolve the issue, after logging in and being redirected back to the React application, several errors are thrown by the BFF service as follows:

The cookie '.AspNetCore.OpenIdConnect.Nonce.......' has set 'SameSite=None' and must also set 'Secure'.

       ---> System.Exception: Correlation failed.
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler`1.HandleRequestAsync()
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)```

[dclayton77]

This is the full config of my BFF service:

using Duende.Bff.Yarp;
using Microsoft.IdentityModel.Logging;
using Platform.Web.Configuration;

var builder = WebApplication.CreateBuilder(args);

var authenticationSettings = new AuthenticationSettings();
var authenticationSettingsSection = builder.Configuration.GetSection(nameof(AuthenticationSettings));
authenticationSettingsSection.Bind(authenticationSettings);

var proxySettings = new ProxySettings();
var proxySettingsSection = builder.Configuration.GetSection(nameof(ProxySettings));
proxySettingsSection.Bind(proxySettings);

builder.Services.AddControllers();
builder.Services.AddBff().AddRemoteApis();

builder.Services.AddAuthentication(options =>
{
    options.DefaultScheme = "cookie";
    options.DefaultChallengeScheme = "oidc";
    options.DefaultSignOutScheme = "oidc";
}).AddCookie("cookie", options =>
{
    options.Cookie.Name = "__Host-bff";
    options.Cookie.SameSite = SameSiteMode.Strict;
}).AddOpenIdConnect("oidc", options =>
{   
    options.Authority = authenticationSettings.Authority;
    options.ClientId = authenticationSettings.ClientId;
    options.ClientSecret = authenticationSettings.ClientSecret;
    options.ResponseType = "code";
    options.ResponseMode = "query";
    options.RequireHttpsMetadata = true;

    options.GetClaimsFromUserInfoEndpoint = true;
    options.MapInboundClaims = false;
    options.SaveTokens = true;

    options.Scope.Clear();
    foreach (var scope in authenticationSettings.Scopes)
    {
        options.Scope.Add(scope);
    }

    options.TokenValidationParameters = new()
    {
        NameClaimType = "name",
        RoleClaimType = "role"
    };

    options.Events.OnRedirectToIdentityProvider = async n =>
    {
        n.ProtocolMessage.RedirectUri = authenticationSettings.RedirectUri;
        await Task.FromResult(0);
    };
});

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    IdentityModelEventSource.ShowPII = true;
}

app.UseForwardedHeaders();

app.UseHttpsRedirection();

app.UseStaticFiles();
app.UseRouting();
app.UseAuthentication();
app.UseBff();
app.UseAuthorization();

app.MapControllers()
    .RequireAuthorization()
    .AsBffApiEndpoint();

app.MapBffManagementEndpoints();
foreach (var apiEndpoint in proxySettings.ApiEndpoints)
{
    app.MapRemoteBffApiEndpoint(apiEndpoint.LocalPath, apiEndpoint.ApiPath).RequireAccessToken();
}

app.MapFallbackToFile("index.html");

app.Run();

[josephdecock]

It looks like your BFF is not configuring any options for the forwarded headers middleware. Without that - even though you've added the middleware to the pipeline - the middleware doesn't do anything.

You need something like this in the BFF:

builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
        ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
});

Without that, the OIDC handler sees the incoming requests as http, not https, which causes it to infer that it should not set the secure attribute on the cookies it sets. That conflicts with the same site attribute that it is setting, causing the cookie to not be sent back to the BFF by the browser. Without the state that the handler roundtrips in this way in the correlation and/or nonce cookies, the handler gives the errors that you're seeing.

[dclayton77]

Thank you for the further advice, that seemed to be what I was missing. However, with those settings added, I still get the same errors. I am wondering if my ingress is not providing the headers so will investigate that.

[abdullah-itblue]

I am currently experiencing the same issue as mentioned by user @dclayton77. Are you also utilizing Traefik as the proxy manager? Kindly inform me if you come across any potential solutions to this matter. I am actively seeking a resolution as well.

[dclayton77]

I am not using Traefik, I have a cloud load balancer that points to a standard ingress controller. I will post back with any findings.

[dclayton77]

After further investigation, I do not believe the problem is with IdentityServer BFF so am closing the ticket. The information here I think will be helpful later in my journey but at present I believe the environment is causing these problems. I am using a Hetzner load balancer, pointing at a Kong Ingress Controller. I also have a LinkerD service mesh. From research I am convinced the problem lies within one of the services in the K8S cluster but I am yet to crack it. Thank you for all the help here, I will re-open the ticket should I still have issues after cracking the environment.

[dclayton77]

I've now reconfigured the K8S cluster so that all services should be passing through the required information and headers. However, I am still facing the issue. I have added some middleware to the BFF service to log out all headers for all requests. When I hit the /bff/login endpoint, I can see that the required headers for protocol etc are present. This is shown in the logs of the running service as follows:

However, I still receive the following error in the BFF service:

The URL is still being set as HTTP. I am obviously still missing something in the BFF service. My full program.cs is as follows:

var builder = WebApplication.CreateBuilder(args);

var authenticationSettings = new AuthenticationSettings();
var authenticationSettingsSection = builder.Configuration.GetSection(nameof(AuthenticationSettings));
authenticationSettingsSection.Bind(authenticationSettings);

var proxySettings = new ProxySettings();
var proxySettingsSection = builder.Configuration.GetSection(nameof(ProxySettings));
proxySettingsSection.Bind(proxySettings);

builder.Services.AddControllers();
builder.Services.AddBff().AddRemoteApis();

builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders = ForwardedHeaders.All;
});

builder.Services.AddAuthentication(options =>
{
    options.DefaultScheme = "cookie";
    options.DefaultChallengeScheme = "oidc";
    options.DefaultSignOutScheme = "oidc";
}).AddCookie("cookie", options =>
{
    options.Cookie.Name = "__Host-bff";
    options.Cookie.SameSite = SameSiteMode.Strict;
}).AddOpenIdConnect("oidc", options =>
{   
    options.Authority = authenticationSettings.Authority;
    options.ClientId = authenticationSettings.ClientId;
    options.ClientSecret = authenticationSettings.ClientSecret;
    options.ResponseType = "code";
    options.ResponseMode = "query";
    options.RequireHttpsMetadata = true;

    options.GetClaimsFromUserInfoEndpoint = true;
    options.MapInboundClaims = false;
    options.SaveTokens = true;

    options.Scope.Clear();
    foreach (var scope in authenticationSettings.Scopes)
    {
        options.Scope.Add(scope);
    }

    options.TokenValidationParameters = new()
    {
        NameClaimType = "name",
        RoleClaimType = "role"
    };
});

builder.Services.AddLogging(builder => builder.AddConsole());
builder.Services.AddTransient<LoggingMiddleware>();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    IdentityModelEventSource.ShowPII = true;
}

app.UseForwardedHeaders();

app.UseHttpsRedirection();

app.UseStaticFiles();

app.UseMiddleware<LoggingMiddleware>();

app.UseRouting();
app.UseAuthentication();
app.UseBff();
app.UseAuthorization();

app.MapControllers()
    .RequireAuthorization()
    .AsBffApiEndpoint();

app.MapBffManagementEndpoints();
foreach (var apiEndpoint in proxySettings.ApiEndpoints)
{
    app.MapRemoteBffApiEndpoint(apiEndpoint.LocalPath, apiEndpoint.ApiPath).RequireAccessToken();
}

app.MapFallbackToFile("index.html");

app.Run();

[dclayton77]

Some further information which may or may not be relevant. My front end site and BFF service are being served on a domain such as www.dev.mydomain.co.uk. My IdentityServer is being served on a domain such as auth.dev.mydomain.co.uk.

Having read a little more into SameSite.Strict setting, I believe sub domains will not work. I have tried changing the cookie options in the BFF service to the following:

    options.Cookie.SameSite = SameSiteMode.None;
    options.Cookie.SecurePolicy = CookieSecurePolicy.Always;

However, after changing this setting and deploying the service, I still have the same problem with the error:

The cookie '.AspNetCore.Correlation.' has set 'SameSite=None' and must also set 'Secure'.

I am not sure if I also need to change this on the IdentityServer or in the React application to match.

@AndersAbel or @josephdecock, do you think my issue could be something along these lines or am I looking in the wrong direction?

[dclayton77]

Closing this issue as it has got quite heavy, will open a new issue with more specific information. The problem is not resolved but this thread seems to have gone stale.

[dclayton77]

Adding a further comment to help others as I have now found the solution.

Thanks to this thread I found the answer. They key is to combine the forwarded headers configuration so that it is set as follows:

var forwardedHeadersOptions = new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.All,
    ForwardLimit = 1
};
forwardedHeadersOptions.KnownNetworks.Clear();
forwardedHeadersOptions.KnownProxies.Clear();
app.UseForwardedHeaders(forwardedHeadersOptions);