403 when accessed behind a load balancer

[sourabhtewari]

Describe the bug

403 response when we access an Identity enabled client behind a load balancer. We haven't added more nodes, so there is only once node behind the ELB which is serving all requests.

To Reproduce

Steps to reproduce the behavior:

We use an Elastic Load Balancer (AWS) for our needs. There are two scenarios. One is access from corp environment and another from public internet. For corp, there is a direct pass through and it goes straight to the host. But accessing from public internet, it uses the load balancer.

In both the scenarios, we can get to the Identity Server and login. But using an Identity enabled client, like Identity Admin, we get a 403 response when accessing from public internet.

In the log below, the same request is from two locations. (1) to (21) is from corporate internet and the response is Authorization Successful. From (22) to (25), its from public internet, the same request results in 403.

Relevant parts of the log file

Id	Message
25	Request finished in 2.132ms 302
24	AuthenticationScheme: "oidc" was challenged.
23	Authorization failed.
22	Request starting HTTP/1.1 GET https://XXXXXXXXXXXXXX/IdentityAdmin
21	Request finished in 2.1554ms 200 image/x-icon
20	Sending file. Request path: '"/favicon.ico"'. Physical path: '"C:\Octopus\Applications\Cloud\XXXXXX.Identity.Admin\2.0.0-IdentityServer4-Admin\wwwroot\favicon.ico"'
19	Request starting HTTP/2.0 GET https://XXXXXXXXXXXXXX/IdentityAdmin/favicon.ico
18	Request finished in 5.1559ms 200 text/html; charset=utf-8
17	Executed endpoint '"XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)"'
16	Executed action "XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)" in 3.684ms
15	Executed ViewResult - view "Index" executed in 3.3473ms.
14	Executing ViewResult, running view "Index".
13	Route matched with "{action = \"Index\", controller = \"Home\"}". Executing controller action with signature "Microsoft.AspNetCore.Mvc.IActionResult Index()" on controller "XXXXXX.Identity.Admin.Controllers.HomeController" ("XXXXXX.Identity.Admin").
12	Executing endpoint '"XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)"'
11	Authorization was successful.
10	Request starting HTTP/2.0 GET https://XXXXXXXXXXXXXX/IdentityAdmin
9	Request finished in 5.5561ms 200 text/html; charset=utf-8
8	Executed endpoint '"XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)"'
7	Executed action "XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)" in 4.0668ms
6	Executed ViewResult - view "Index" executed in 3.6788ms.
5	Executing ViewResult, running view "Index".
4	Route matched with "{action = \"Index\", controller = \"Home\"}". Executing controller action with signature "Microsoft.AspNetCore.Mvc.IActionResult Index()" on controller "XXXXXX.Identity.Admin.Controllers.HomeController" ("XXXXXX.Identity.Admin").
3	Executing endpoint '"XXXXXX.Identity.Admin.Controllers.HomeController.Index (XXXXXX.Identity.Admin)"'
2	Authorization was successful.
1	Request starting HTTP/2.0 GET https://XXXXXXXXXXXXXX/IdentityAdmin

[sourabhtewari]

The issue is due to a WAF rule GenericRFI_QUERYARGUMENTS

This blocks implicit grant requests. Any ideas to circumvent this?

[skoruba]

Hi @sourabhtewari - did you find the issue? Thanks!

[sourabhtewari]

@skoruba Found the issue but not the solution . WAF blocks any call with "://" pattern in the query parameters. More info for the WAF (RFI related rules) here https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-list.html

Asked a question https://github.com/IdentityServer/IdentityServer4/issues/5231.

[sourabhtewari]

Opening the issue just @skoruba could help with a solution :)

[sourabhtewari]

@skoruba we have logged a ticket with AWS who will help us set up the WAF. this has nothing to do with Identity. Closing the ticket.

[skoruba]

@sourabhtewari - Thanks for your feedback, I do not have any experience with this issue and AWS in general. 😉

[apetrut]

@sourabhtewari Did the STS and Admin services communicate over internal DNS names when deployed in AWS? Or did you use public DNS names?

[sourabhtewari]

Use external dns for prod and internal for pre prod. The 403 was due to WAF ACL blocking the requests, which we sorted it out.

[apetrut]

@sourabhtewari If we use internal DNS we get this flow in the browser:

htc-identity-sts is the internal DNS but it should be sts.mydevurl.com

[sourabhtewari]

Apologies, I should have been more clear. We use OpenShift. The service is exposed as a Kubernetes route. The dns for our preprod is internal to our local corp. All our environments are sts..com