missing forwareded headers

[Electrofenster]

when using the oidc plugin a few forwarded headers got missing.

Without using the oidc plugin I get this:

X-Forwarded-Host: whoami.domain.tld
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: a228d1c86e24

when using the plugin the only forwarded header I got is this: X-Forwarded-For: xx.xx.xxx.xxx

[travisghansen]

I assume you’re referring to which headers make it all the way to your backing service instances?

Which proxy do you use? Can you how you have it configured to communicate with eas? Can you send over the config token (remove sensitive data)?

[Electrofenster]

Yes, I mean the headers which are needed for my backing service instances.

I use traefik.

let config_token = {
    eas: {
        plugins: [
            {
                type: "oidc",
                issuer: {
                    discover_url: "http://keycloak.local/auth/realms/xxxx/.well-known/openid-configuration"
                },
                client: {
                    client_id: "external-auth",
                    client_secret: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
                },
                scopes: ["openid", "email", "profile"],
                custom_authorization_parameters: {},
                features: {
                    cookie_expiry: true,
                    userinfo_expiry: true,
                    session_expiry: true,
                    session_expiry_refresh_window: 86400,
                    session_retain_id: true,
                    refresh_access_token: true,
                    fetch_userinfo: true,
                    introspect_access_token: true,
                    introspect_expiry: 0,
                    authorization_token: "access_token"
                },
                assertions: {
                    exp: true,
                    nbf: true,
                    iss: true,
                },
            }
        ],
    }
};

the minimal traefik setup looks like this:

http:
  middlewares:
    external-auth:
      forwardAuth:
        address: "http://external-auth:8080/verify?config_token=x2%2FzWjjMkQNq%2F0mXbn%2Fb8UGnuDBo%2FLBwiJZQRDzquwMFiFtJ0sYawlGqBq%2BHzxT%2BtM14DPZBNNFQF1ifEI%2Bgq6kGLP6FS%2Bv%2F%2BHyuG%2BnjNfDDmyduf4Qq646dNcG3gDGJLig8rI7KUjAe%2BSHjUamSdRO%2FsJylMTSUor1xFnz5MVrj1f21Vl6qAXDS1mTVyaSCYq6bAZsyj4vG3o%2BAozKBJvRbBExZLkKYSNb%2BcPmVP9y7k%2FbjaXS8m753VhV%2BaH%2FSz%2FFXaI9i6BzdXSt7OcDENSzGx5D0Xc7JbtNqTCxSN9CsNVY1LEmgAhdnfuM9LJQYGfrp3nCyBjw4fQv0xkyUOv8o1yjop0l%2FHE1vBc8KEb4FP%2Fe2TOYEJA3nSP7naTCFUMWLmJoy%2B%2FAKA1C5sCMOfwrueak9GT0khVbRw9HQOf8kpkDz7SR8Ma5697Vsp4VXfzUmyxJn%2BeR%2Bigv3Cx6HAOadlrX7n4G6iODuY15yY9oWXS6qg5uQE0kpGsdLo3fpTzcxNGB8J3gMH9V78GVtkU%2BihrESH15kAlHxf3DDDrAWMEFzMxKphiyz1EwAH7y%2FnJn3Wx6LKnCnVvXd2JRipiGPl0m47Hbkr6EW1AydMQqKjR0Mp4ai1ERPjdpBgwDeD28xat2zcwOxV8EvIHJjIO2QojYIFEzS8u14llePqGKu8vGa%2BHsivl8QhopBSU%2FNeFzhEiLAsYvBZyiOWe6ic8T%2BDszedA6%2FGg3r0zPqz4kyxeRguwHeoFlqp2ma2UXm0v%2F0vdXEI2lCD0xsJAaU4S6xRo5pubUUFEHywXhH3TRSnTazlzDDq7rvLIvQxUO2f2Kb22m2dF6K3nBsU0ihPydo%2FYMsE%2FaYmd%2B0qptk1yX0Mlm2%2B3SrDkHnxSK87R9ya%2B3GlrbQHMaHvN6aFF68oiZqEzbXhsUj1lDvWv4iU2q4O7mbwxIW%2FvWD%2FNc1exVlE22N5fxWVTSBZFCatUdJPjE%2BV4eOp0AbmqYx2lu5kXeKQPhUl%2B8OrThUJfGt4CheRoA%2ByXTieoOI6p9%2BFyoJfOWXdpEjBpF%2BEPX6sMGOF%2BomgYJe1k2bjH99g2ugH1boTWv1PZ4zGYn8djGkOvtZCiPfBnPjAymUMBgtzruEdeRTJSfQV1wh8WKbaEVeh0fYgis6sCV1nHQ%2B%2FtER0BrE%2Fm50VZZllRfx7T0iILD5AoqQHrwehYeJDY1t9PKKPRy3zw3STmZJmCwyq4U9uJcMWp1PuLCUHVR5KbAGWTbWu3VcBw63PtR7SCRV3Mq1tn8p"
        authResponseHeadersRegex: "^X-"
        trustForwardHeader: true

  services:
    whoami:
      loadBalancer:
        servers:
          - url: "http://whoami:80"

  routers:
    whoami:
      rule: "Host(`whoami.local`)"
      service: whoami
      middlewares:
        - external-auth
      entryPoints:
        - http

before I updated to a newer version, the missing headers were present.

[travisghansen]

Seems like a bug/misconfigured traefik setup.

Can you try the authResponseHeaders instead of the regex setup and explicitly call out what you want passed down?

Also what exact version of traefik are you running? I may get a chance to try and reproduce the issue.

[Electrofenster]

Yeah, I use traefik v2.4.8. I just added these lines:

authResponseHeaders:
  - "X-Forwarded-Host"
  - "X-Forwarded-Port"
  - "X-Forwarded-Proto"
  - "X-Forwarded-Server"
  - "X-Access-Token"
  - "X-Userinfo"
  - "X-Id-Token"

and I only get:

  - "X-Access-Token"
  - "X-Userinfo"
  - "X-Id-Token"

when I remove external-auth from middlewares, I'll get the missing headers.

[travisghansen]

Yeah that’s expected (roughly). The parameter is meant to tell the proxy which headers returned from the auth service do you want sent down to the backing service. eas does not set the X-Forwarded- headers when responding back to traefik.

It sets the 3 you mentioned and according to your config token is also setting Authorization to the access_token value.

In short, set your headers parameter to only the 3 mentioned (assuming you want those passed on) and possibly Authorization. If you don’t want any of those passed down leave it blank.

That should solve your issue. Let me know how it goes.

[Electrofenster]

My services behind traefik and eas require these headers:

• "X-Forwarded-Host"
• "X-Forwarded-Port"
• "X-Forwarded-Proto"
• "X-Forwarded-Server"

I don't know where I should configure it to pass these headers down to my services. It worked before the last update where you added introspection. And for now, I don't understand where I should configure it.

[travisghansen]

Are you certain you’ve got this configured?

https://doc.traefik.io/traefik/v2.1/routing/entrypoints/#forwarded-header

[Electrofenster]

No, i don't have this configured. But I don't understand why I should configure it?

When I remove your middleware from the router I got these missing headers. I don't understand why I got these without your middleware but not when using your middleware.

But when I configure it, I also don't get these headers :/

[travisghansen]

Did you try this?

authResponseHeaders:
  - "X-Access-Token"
  - "X-Userinfo"
  - "X-Id-Token"

[Electrofenster]

well... that worked?! When I use this I got:

• X-Access-Token
• X-Forwarded-For
• X-Forwarded-Host
• X-Forwarded-Port
• X-Forwarded-Proto
• X-Forwarded-Server
• X-Id-Token
• X-Real-Ip

but can you explain why?

[travisghansen]

That's what I was trying to suggest above at first. You're telling traffic to forward the given headers from the auth server to the backend service. The auth server (eas) is not sending X-Forwarded-Foo headers in it's response to traefik, so traefik is trying to replace those being sent to the backing service as set by eas...which is empty/not there.

[travisghansen]

The regex behavior is potentially a bit weird FYI. I would expect the regex to not replace all headers in the original request passing the regex but rather would expect it to be a sort of allow list treating those that are returned by the auth service to be set to whatever the auth service responds with.