Mapping bypass_auth not always honored

[carlin-q-scott]

Describe the bug I have an AuthService with a Mapping that has bypass_auth set to true. Sometimes a request to auth service goes through AuthService instead of the Mapping.

The situation where we're frequently seeing this is:

1. Browser requests /app without jwt cookie
2. ext_authz request sent to /auth
3. Auth returns 302 to /login
4. Login returns 302 to /auth/callback
5. Auth is sent an ext_authz request instead of the full browser request
6. The OAuth callback fails because the Content-Type header is missing
7. If we retry the request from the browser, it usually succeeds and our Auth service is sent the full, non ext_authz request. But sometimes we can retry up to 5 times before it goes through correctly

Our Auth service manifest:

apiVersion: getambassador.io/v3alpha1
kind:  Mapping
metadata:
  name: auth
spec:
  host: "*"
  prefix: /auth/
  service: http://auth.default:80
  bypass_auth: true
  rewrite: /auth/
---
apiVersion: getambassador.io/v3alpha1
kind: AuthService
metadata:
  name: auth
  namespace: default
spec:
  auth_service: "auth.default:80"
  proto: http
  timeout_ms: 5000
  include_body:
    max_bytes: 4096
    allow_partial: true
  status_on_error:
    code: 502
  failure_mode_allow: false

To Reproduce Steps to reproduce the behavior:

1. Configure an auth service like above with request logging
2. Log in through that auth service
3. You'll likely see that the auth service was sent an ext_authz request on the callback redirect

Expected behavior The full browser request makes it to the callback endpoint on the auth service

Versions (please complete the following information):

• Ambassador: 3.3.0
• Kubernetes environment: Azure Kubernetes Service
• Version: 1.24.6

Additional context

Comparison of original request (authz) and browser refresh:

Workaround Allow the accept and content-type header to be passed through to the authz endpoint by adding this to the AuthService definition:

  allowed_request_headers:
    - accept
    - content-type

[cindymullins-dw]

Hi @carlin-q-scott, thanks for raising this issue and the workaround. I wouldn't think ext_authz would be interfering if bypass_auth is used but perhaps there is some lag/ordering issue in chain of events. Also found this Envoy issue which may also be related.

[carlin-q-scott]

@cindymullins-dw Yeah, looks like they implemented the same workaround as me. They were also using emissary-ingress.

I don't think I can give much input on the lag/ordering you mention since I don't know the internal workings of emissary and envoy. Are the request chains executing in parallel and first one to respond wins? So when I make a request to my bypass_auth Mapped service, emissary goes ahead and makes a request to the AuthService and would cancel that request if the target service responds faster? One theory I had was that emissary or envoy is reusing the request forwarder for the AuthService when fulfilling the request to the Mapped service since they are in fact the same service.