Auth Request Redirect annotation does not behave properly

[TobyTheHutt]

What happened: I am trying to fix an issue for a jira instance of mine, regarding redirects. When a user logs in for the first time without a valid SAML assertion, they get redirected to their IdP and once they come back, they land on the start page, rather than the resource they called (their Kanban board or whatever). The instance is protected by an OAuth2 proxy.

I was able to fix this with a simple config snippet:

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header X-Auth-Request-Redirect $request_uri;

While this solution works fine enough for me, it bugs me that I cannot use the official annotation:

nginx.ingress.kuberentes.io/auth-request-redirect: $request_uri

When I use the nginx.ingress.kuberentes.io/auth-request-redirect annotation, the redirect stops working and I land on the Jira start page again after authentication. I tried to supply the $request_uri variable plain like this and also with ${request_uri} but both versions do not change this behaviour.

What you expected to happen: I would expect that these two options are just two sides of the same coin. So either way of configuring it should result in the same behaviour.

Environment:

Since I'm not the cluster admin of the Kubernetes instance I cannot give detailed informations on the Nginx version or the Controller. This is the currently working Ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Auth-Request-Redirect $request_uri;
    nginx.ingress.kubernetes.io/proxy-body-size: 750m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
  labels:
    app.kubernetes.io/instance: myjira
    app.kubernetes.io/name: myjira
    app.kubernetes.io/version: 9.4.7
  name: myjira
spec:
  ingressClassName: nginx
  rules:
  - host: myjira.example.com
    http:
      paths:
      - backend:
          service:
            name: jira
            port:
              number: 80
        path: /jira
        pathType: Prefix

When I analyze what happens in the network, the difference between the two configurations is already eminent in the first request.

This is my working web call to my Jira instance with the configuration snippet I mentioned above. It's already visible that the redirect URL got registered and is present in the answering web call:

This on the other hand is what happens, if I remove the configurations snippet from the ingress, or replace it with the nginx.ingress.kuberentes.io/auth-request-redirect: $request_uri annotation:

We can see that in the second response, there is only a percent-encoded %2F which is just the / or the root of my application, which leads to the start page.

I cannot currently explain to myself how this behaviour comes to be, so I hope someone here can help me. If further information is needed, let me know.

[longwuyuan]

Is it possible that the Auth service needs to set the URL to which you are redirected ?

[TobyTheHutt]

Not sure whether you mean the OAuth2 proxy or my SSO provider (KeyCloak) when you say Auth service, but I am mostly sure that the issue isn't there.

My point is, that the Nginx Ingress behaves differently when I set the X-Auth-Request-Redirect header directly as a configuration snippet (so as plain Nginx config), or when I set the nginx.ingress.kuberentes.io/auth-request-redirect annotation. The annotation was proposed specifically for this purpose in the thread #1979 and implemented with #1993. To my understanding, the two should behave the same, which they don't.

To the proxy header itself As far as I understand it, the X-Auth-Request-Redirect header does nothing more than set a header containing the original request URL for the upstream proxy (in my case, that'd be the OAuth2 proxy), so the user is properly redirected after successful authentication. Though I'm having a hard time finding proper documentation on that header, so I might be only partially right on this.

As I see it, it makes sense that the downstream proxy which first handles the request sets this header for all upstream proxy instances behind it. In my scenario that would be the Nginx Ingress.

[longwuyuan]

in that case please post both nginx.conf as attachments here or even better just copy/paste the related lines from both the resulting nginx.conf files.

If someone can reproduce, this could be a potential bug. But for that a precise detailed reproduce procedure needs to be available . Because it is a auth issue, it will be tasking for a reader to reproduce by themselves implementing a auth server. The image httpbun.com is a good prospect for testing auth but work is needed to make it relevant in this issue reproduce

[TobyTheHutt]

I was able to debug the issue some more on a local environment. Please keep in mind that this issue is not about authentication services, but about the X-Auth-Request-Redirect HTTP header, which does not seem to take effect when the official Ingress annotation is used. The behaviour regarding this issue can already be observed in the initial request of any web call, regardless of whether static web content or an authentication service is called. The implementation of an authentication service was therefore not deemed necessary. Still, I can quickly implement these components as well, should this be considered relevant for the case.

TL;DR: I'm able to reproduce the issue with a fresh Setup with the current Ingress Controller release 1.9.5. The X-Auth-Request-Redirect header only takes effect when using the config snippet. Using the designed Ingress annotation has no effect on either the configuration or the resulting HTTP headers.

Environment

• Kubernetes Version: 1.27.2
• Kubectl Version: 1.27.1
• Helm Version: 3.12.0
• Nginx Controller Version: 1.9.5
• Httpbun Version: 95f529281db0ee63b150ea3877b1f29671cf4f0f

Setup

You will see below that the configuration is quite minimal, which should make it easy to reproduce the behaviour in other enviroments as well.

httpbun.yml

#httpbun.yml mentioned below
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: httpbun
  name: httpbun
  namespace: ingress-debug
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbun
    spec:
      containers:
      - args:
        - --path-prefix=httpbun
        image: sharat87/httpbun
        name: httpbun
        ports:
        - containerPort: 80
          name: httpbun
          protocol: TCP

Install procedure

# Create new K8s namespace
kubectl create ns ingress-debug

# Deploy Httpbun and Svc
kubectl apply -f httpbun.yml
kubectl expose deploy httpbun

# Install & create Nginx Ingress
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
kubectl create ingress ingress-debug --class=nginx --rule="ingress-debug.localdev.me/*=httpbun:80"

Tests

All tests use the same debugging URL below. URL: http://ingress-debug.localdev.me/httpbun/headers

Test 1: Reproduction of working config snippet

Ingress annotation:

nginx.ingress.kubernetes.io/configuration-snippet: |
   proxy_set_header X-Auth-Request-Redirect $request_uri;

Result:

Nginx config:

http {
  ...
  server {
    ...
    location /httpbun/ {
      ...
      proxy_set_header X-Auth-Request-Redirect $request_uri;
      ...
    }
  }
}

Test 2: Experiments with Ingress Annotation

Used Annotations:

• nginx.ingress.kuberentes.io/auth-request-redirect: $request_uri
• nginx.ingress.kuberentes.io/auth-request-redirect: ${request_uri}
• nginx.ingress.kuberentes.io/auth-request-redirect: /httpbun/headers

Result: In each of the 3 variations, only the Request ID changed, but the X-Auth-Request-Redirect could never be produced.

The necessary Nginx config to actually send the header was also missing.

Conclusion

The Ingress Annotation nginx.ingress.kuberentes.io/auth-request-redirect does not work, or only works in specific scenarios which are not sufficiently documented.

However, the Configuration Snippet mentioned above is a valid workaround.

⚠️ It is still important to note, that enabling the configuration snippet raises other security concerns, as documented in the annotations.md

The Nginx file with and without configuration snippets can be found here: nginx_conf.zip

The file only changes with the configuration snippets. The presence or absence of any of the tried nginx.ingress.kuberentes.io/auth-request-redirect annotations did not cause the configuration to differ.

[longwuyuan]

@TobyTheHutt thank you very much for the reproduce procedure. This makes it easier for a developer to look at this issue

cc @tao12345666333 @rikatz @Gacko

/triage accepted

[longwuyuan]

@TobyTheHutt does this example work for you https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/ ?

[TobyTheHutt]

Thank you for the article.

The required endpoints are found and reached without the two Annotations mentioned in the article you provided. Users always get properly redirected from the OAuth2 proxy to the IdP. Please keep in mind: My issue is not the authentication, nor the redirect to my IdP provider/broker. The issue at hand here is solely the behaviour of the nginx.ingress.kuberentes.io/auth-request-redirect Ingress annotation which seems to have no effects at all to the behaviour of the Ingress, according to my analysis.

Of course, to validate your proposal, I also ran a few tests.

⚠️ Please note that a lot of the following analysis has nothing to do with the Nginx Ingress, but with OAuth2 proxy and its behaviour. Therefore, I also didn't setup a dev environment with an Oauth2 proxy and an IdP for this. Instead, I will use foobar names as URI references. It is still assured that every statement has been verified on a working testing environment.

If I set the headers like in the example below, the necessary X-Auth-Request-Redirect header is still missing. Without this header, I still land on the start page (/) after a successful logon. Note that the endpoints /start and /auth are Oauth2 proxy endpoints and have nothing to do with Jira.

nginx.ingress.kubernetes.io/auth-signin: https://$host/jira/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: http://myjira-svc.myjira-namespace.svc.cluster.local/jira/auth

I also tried to add the rd option on the auth-url annotation and I tried the escaped as well as the non-escaped variable name - without success.

However, if I also add the already known and discussed config snippet implementing the X-Auth-Request-Redirect header from this post, everything from the initial logon to the final redirection to my requested resource works. This makes the two headers discussed here obsolete, since everything works just as well without them (at least in my setup).

Conclusion The OAuth2 Proxy project also does not work fully as desired, since the rd query string should be handled with the highest priority, since it's also the most specific one, according to their own code documentation. I will also create an according issue in their repo these days.

However, even with the Annotations provided by your link, neither an additional header is supplied, nor does the behaviour of the whole logon process change.

[longwuyuan]

Just FYI, successful auth responds with requested URL but the header you mentioned is not included for sure

[TobyTheHutt]

Everything that happens with these annotations auth-url and auth-signin is completely up to the backend and in no way in the responsibility of the Nginx Ingress.

Regarding the auth-url annotation The auth-url annotation only sets an URL, against which each request should be verified. That's not in itself a redirect, it's just an additional web call that the ingress makes to verify each request. It does that by forwarding the request information to the URL specified. If the auth-endpoint recognizes known and valid authentication data (or simply responds with a positive HTTP status), the actual called endpoint or page can be reached.

Example In the case documented by the Ingress Nginx docs, the /oauth2/auth endpoint is called for the auth-url annotation. This endpoint just responds with a status 401 if the sent authentication data is valid and a status 202 if valid authentication data was sent. If the Ingress receives any reply with a status 400 or higher, it's not really documented what happens next, but likely the request is forwarded to the URL specified in auth-signin.

No headers get added or modified in either of the two annotations.

[TobyTheHutt]

Sidenote: The Annotations auth-url and auth-signin are meant to be used in combination. As mentioned in my last comment, auth-url has the purpose of forwarding requests without valid auth-information to the auth-signin endpoint. Using either without the other would, as I see it, not result in any meaningful configuration. In the auth-signin endpoint we can also implement redirect logic, so I could for example append the option rd=$escaped_request_uri to the defined endpoint, which is prioritized even higher than the X-Auth-Request-Redirect header.

To get back to your point @longwuyuan: Using auth-url and auth-signin would be a valid alternative to my current workaround with the configuration-snippet annotation. But they don't work, because they result in a Proxy error 503 when the auth-url endpoint returns a status higher than 400.

This was already documented in #8401 but the issue was closed due to its rotten state.

[TobyTheHutt]

One last thing: I created a debug-log to analyze the main topic from this ticket some more (the auth-request-redirect annotation): ingress-debug.zip

In there I see that my hardcoded value /httpbun/headers for the auth-request-redirect annotation is present, but somehow gets ignored anyway. If required I can also upload the full log without the filter on the Ingress resource.

Sorry for the spam, I'll stay put for your feedback for now.

[longwuyuan]

I don't have the knowledge or skills to make conclusive remarks on the destination URL after auth, via the redirect. The header X-Auth-Request-Redirect is missing in my test too.

There is acute shortage of dev resources so please wait for comments from others and developers.

[TobyTheHutt]

I found out the root cause for the missing header.

Nginx sub-locations

When nginx.ingress.kubernetes.io/auth-url is defined, the Nginx controller creates corresponding sub-locations in its nginx.conf which are then linked to the actual Ingress location by the auth_request option. This happens, because the auth-url annotation enables external authentication.

Who sends and receives what?

Now, when the user starts a request, the Ingress first sends an additional request to the URL defined in auth-url, as I suspected in an earlier comment. But this does not go out directly from the location that I use as my application context (which would be "/httpbun"), but it goes out from the created sub-location which was specifically created for this purpose. Here is a screenshot from my TCP dump:

As you can see, here's my pretty header that I was looking for so hard. I called "/httpbun/headers" but the Ingress "secretly" forwarded my request to the defined auth-url so it could properly verify that I'm allowed to access this resource.

Conclusion

In the scenario of external authentication, the nginx sub-location does the authentication and therefore sets the authentication headers. Logically speaking, this is another, separate proxy instance which handles my authentication requests. This is why I never saw the header in httpbun - because the Nginx sub-location was the caller for the authentication request, not me.

When I set the X-Auth-Request-Redirect header with the configuration-snippet annotation, my input is not interpreted by the ingress and is therefore directly persisted in my Ingress location, and not in a sub-location for external auth. There, the header is appended directly to my Request, where I can also see it in the httpbun "/headers" interface.

The documentation sadly doesn't support a deeper understanding of this process. The logic how the Ingress behaves in the scenario of external authentication, what the workflows look like ,which logical components are involved and what their dependencies are is only sparingly described in a few text blocks and bullet points.

Next steps

I will run a few more tests and get some tcp dumps from the Ingress controller itself to understand better how I can successfully implement my specific business case in this setup. If I find a better solution than using the config snippet provided in my initial message, I will post it here.

In the course of my analysis, I also found another bug for which I'm not sure what the gravity of it is: The sub-locations contain the base64 string of the original location. If the original location has no trailing slash in its path, the base64 string becomes corrupt. In my case, "/httpbun/" would correctly have the base64 string "L2h0dHBidW4v", whereas "/httpbun" would result in "L2h0dHBidW4", which is the same base64 string but with the last character missing. The references within the configuration are correct though. I will create a new issue for this.

What is your point of view regarding the documentation? I know from browsing through the issues that I'm by far not the first one with this problem and some more workflow-documentation could potentially do a lot for people with similar issues.

[longwuyuan]

I personally have to read and learn about what a sub-location is. Then I need to understand how/why the request to the external auth-url did go out from the sub-location.

Thank you very much for the debug info. Helps makes a lot of progress. If you want to submit docs PRs, I think they will be very welcome.

We need to wait for reduced load and time availability from others.

Once again, thanks for detailing the problem. It will e interesting to get and read the tcpdump, in the way you described.

[TobyTheHutt]

Just a quick documentation for reference: ingress-debug.zip In there you find:

• the httpbun deployment YAML with the netshoot sidecar container (access with kubectl exec -it <pod> netshoot -- /bin/zsh)
• the TCP dump from the tcp interface of my httpbun pod
• The Nginx configs with and without the auth-url annotation (I also set the auth-request-redirect annotation, just to show the header is correctly present)

To create the tcpdump, I ran tcpdump -i eth0 -w dump.pcap which I could then download from the container and load the file in my Wireshark client to analyze.

Regarding sub-locations, this is just a logical construct. It's basically just another Nginx location, like a vhost in Apache. What makes it a sub-location is just the fact that its purpose is to be accessed by another Nginx location from the same instance.

How to read the data

I always called the same URL for each request: http://ingress-debug.localdev.me/httpbun/headers

TCP dumps The tcpdumps can be opened with Wireshark for a prettier view of the data. Of course, tcpdump as a command can also used if preferred.

The following IPs are involved:

• 10.1.0.17: My Ingress IP
• 10.1.0.19: My httpbun/backend IP
• 192.168.65.4: My clients IP address

In the first request, everything is straight-forward (WS query tcp.stream eq 0):

1. One call from the Ingress to the backend (with my client IP in the X-Real-IP and X-Forwarded-For headers)
2. One response from the backend with the JSON data from the httpbun /headers endpoint

In the second request, there is more happening (WS queries tcp.stream eq 0 and tcp.stream eq 1): You can already see that there are two TCP streams opened, since we have basically two separate forms or threads of this request.

1. A request is opened for "/httpbun/basic-auth/username/password" even though I never called this URL 1.1. That's because this is the URL stored in the auth-url annotation, which takes effect now and results in a new, initial request 1.2. Note that the initial request already contains the "X-Auth-Request-Redirect" header
2. My client receives a 401 because authentication is missing, which basically ends the first TCP stream 2.1. This results in a Basic Auth prompt in my browser
3. My second attempt goes again the same route, but this time with the "Authorization" header, containing my Basic Auth credentials
4. I get a temporary redirect (307) which allows me to call the "/httpbun/headers" endpoint which I wanted to call

nginx.conf Best you compare the two files in a VS Code, notepad++ or whatever editor tickles your fancy. You see immediately when comparing, that new Nginx locations were defined: "/_external-auth-L2h0dHBidW4v-Prefix" and "/_external-auth-L2h0dHBidW4-Exact"

These locations are then referred in their respective parents (don't know the proper term, but I mean the location they originate from) in the auth_request option. This is why they are called sub-locations - they are not meant to be called from outside, but from another existing nginx location.

Next Steps

I found out why the base64 string was corrupted: template.go We remove the base64 padding (=) for unknown reasons. If no padding is needed for the base64 string, it works. For example, "L2h0dHBidW4v" becomes "/httpbun/". If a padding is needed, the string gets corrupted because "L2h0dHBidW4=" translates to "/httpbun" and without the padding, its not valid base64.

I will create a PR to fix this and I will also consider providing Doc PRs if I find time. When anyone else sees this and feels equal to the task, you're also welcome to it.

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[Gacko]

/assign

[codescalar]

+1 for the docs being very sparse on the concepts and request flow. Many thanks to @TobyTheHutt for the clear investigation notes.

[xXluki98Xx]

Hey, what is the current status on this Issue? I need to set the Redirect header to be able to handle the logout via oauth2proxy and zitadel.