Traefik is removing Access-Control-Allow-Origin headers from the service response

GiamBoscaro commented 8 months ago

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I have a Node service that manages the cors settings (using cors middleware). I am getting a CORS error from our frontend when calling this node service passing through Traefik. Since I haven't had this problem in the past when using nginx or caddy, I was wondering if Traefik was the culprit. I have done some testing.

Calling the API from inside the docker container:

curl --head -X OPTIONS \
  -H "Access-Control-Request-Method: GET" \
  -H "Origin: https://my-domain" \
  http://localhost:3000/info/version

The response I get is this, proving that the cors middleware is working.

X-Powered-By: Express
Access-Control-Allow-Origin: https://my-domain
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Headers: Content-Type,Authorization,Accepts,Set-Cookie,Cookie,Range
Access-Control-Expose-Headers: Accept-Ranges, Content-Encoding, Content-Length, Content-Range
Content-Length: 0
Date: Wed, 14 Feb 2024 09:29:17 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Now, retried using the external URL, passing through Traefik:

curl --head -X OPTIONS \
  -H "Access-Control-Request-Method: GET" \
  -H "Origin: https://my-domain" \
  https://my-domain/info/version

The response is different. It is missing the Access-Control-Allow-Originheader.

access-control-allow-credentials: true
access-control-allow-headers: Content-Type,Authorization,Accepts,Set-Cookie,Cookie,Range
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-expose-headers: Accept-Ranges, Content-Encoding, Content-Length, Content-Range
alt-svc: h3=":443"; ma=2592000
date: Wed, 14 Feb 2024 09:30:00 GMT
vary: Origin
x-powered-by: Express

I was wondering if our corporate proxy was doing something, since I see that also all the headers have been rewritten in lowercase, so I tested a similar call to another service that not using Traefik nor any other proxy but the corporate proxy. Indeed the headers where lowercase again, but Access-Control-Allow-Origin was there. This feels like is Traefik that is doing something to the header.

What I done is then configuring CORS directly with Traefik:

 labels:
      # ...
      # CORS
      - "traefik.http.middlewares.cors.headers.accesscontrolalloworiginlist=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolallowheaders=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolexposeheaders=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolallowmethods=*"
      - "traefik.http.middlewares.cors.headers.accesscontrolallowcredentials=true"
      - "traefik.http.middlewares.cors.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.cors.headers.addvaryheader=true"

In this case, everything work. I can also replace * with something more restrictive. The thing is, what if I wanted to continue managing the CORS from within the Node service instead of Traefik. Because in this situation I need to maintain double the code: the Traefik configuration and also the Node code. I could remove all CORS settings from Node, but what if in the future we change proxy, or if the microservice will be deployed to another environment without Traefik?

Is there a reason why Traefik is removing the header from the response? Is there a way to delegate the CORS management to the service and let Traefik forward all the CORS headers incoming from the service?

What did you see instead?

The response is missing the Access-Control-Allow-Originheader.

access-control-allow-credentials: true
access-control-allow-headers: Content-Type,Authorization,Accepts,Set-Cookie,Cookie,Range
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-expose-headers: Accept-Ranges, Content-Encoding, Content-Length, Content-Range
alt-svc: h3=":443"; ma=2592000
date: Wed, 14 Feb 2024 09:30:00 GMT
vary: Origin
x-powered-by: Express

What version of Traefik are you using?

2.11

What is your environment & configuration?

global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

experimental:
  http3: true

serversTransport:
  insecureSkipVerify: true

ping: {}

log:
  level: DEBUG # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: json # common, json, logfmt
  filePath: /var/log/traefik/out.log

accessLog:
  format: common  # common, json, logfmt
  filePath: /var/log/traefik/access.log

entryPoints:
  web:
    address: ":80"

  websecure:
    address: ":443"
    http3: 
      advertisedPort: 443

  dcv-tcp:
    address: ":8443"

  dcv-udp:
    address: ":8443/udp"

http:
  routers:
    dashboard:
      rule: Host(`traefik.my-domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      service: api@internal
      tls: {}
      entryPoints:
        - websecure
  middlewares:
    traefik-stripPrefix:
      stripPrefix:
        prefixes:
          - "/traefik"

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /run/secrets/tls_cert
        keyFile: /run/secrets/tls_cert_key

providers:
  docker:
    exposedByDefault: false
    network: web
  file:
    directory: /etc/traefik

version: '3.8'

networks:
  public-stage:
    name: public-stage
    driver: overlay

secrets:
  tls_cert:
    external: true
  tls_cert_rsa_key:
    external: true
  tls_root_ca:
    external: true

services:

  traefik:
    image: traefik:v2.11
    hostname: "traefik-{{.Task.ID}}"
    networks:
      - public-stage
    ports:
      - 80:80
      - 443:443/tcp
      - 443:443/udp
      - 8443:8443/tcp
      - 8443:8443/udp
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/media/sambashare/traefik/traefik.yml:/etc/traefik/traefik.yml:ro"
    secrets:
      - source: tls_cert_rsa_key
        target: tls_cert_key
        mode: 0600
      - source: tls_cert
        target: tls_cert
        mode: 0600
      - source: tls_root_ca
        target: tls_root_ca
        mode: 0600
    healthcheck:
      test: ["CMD", "traefik", "healthcheck"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 5s
    deploy:
      mode: global
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 10
        window: 120s
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.25'
          memory: 128M

If applicable, please paste the log output in DEBUG level

No response

mmatur commented 8 months ago

Hi @GiamBoscaro

Thanks for your issue.

Could you please provide us a simple reproductible use case without usage of the corporate proxy?

GiamBoscaro commented 8 months ago

Hi @GiamBoscaro

Thanks for your issue.

Could you please provide us a simple reproductible use case without usage of the corporate proxy?

I will try to setup a working example without the proxy when I have time. Do you think it is something related to the corporate proxy? It is strange because any other service or website doesn't have this problem. Would be really great to know how Traefik is supposed to work also, clearly understanding this would be already a big help for me: if I do not configure any cors middleware in Traefik, should Traefik just forward the response headers coming from the backend service to the client, including cors headers (in particular allow origin) ?

GiamBoscaro commented 8 months ago

Hi @GiamBoscaro

Thanks for your issue.

Could you please provide us a simple reproductible use case without usage of the corporate proxy?

Good morning, an update on the situation. I have have done some testings with the corporate proxy, and I have figured out that my requests are NOT passing through the corporate proxy, since the domains that I am using in the internal network. This means two things:

It is Traefik that it is rewriting the headers lower case
It must be Traefik that is somehow not forwarding the allow origins header in the response.

This is the response when CORS middleware is NOT set:

access-control-allow-credentials: true
access-control-allow-headers: Content-Type,Authorization,Accepts,Set-Cookie,Cookie,Range
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-expose-headers: Accept-Ranges, Content-Encoding, Content-Length, Content-Range
alt-svc: h3=":443"; ma=2592000
date: Mon, 19 Feb 2024 09:21:00 GMT
vary: Origin
x-powered-by: Express

This is the response when CORS middleware is set:

access-control-allow-credentials: true
access-control-allow-headers: Content-Type,Authorization,Accepts,Set-Cookie,Cookie,Range
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-allow-origin: https://my-domain.com
access-control-max-age: 600
alt-svc: h3=":443"; ma=2592000
content-length: 0
date: Mon, 19 Feb 2024 09:24:35 GMT

kevinpollet commented 7 months ago

Hello @GiamBoscaro,

I tried to reproduce the issue with a Go backend and an Express one but without any success. The Access-Control-Allow-Origin header is not removed by Traefik and is forwarded as is to the Client (without using the CORS middleware in Traefik).

Could you please provide a reproducible use case to help us diagnose the issue?

Allypost commented 6 months ago

I had the same issue (docker image traefik:2.11).

It manifested for me when sending an OPTIONS request with access-control-request-method: OPTIONS and origin: https://ANYTHING headers.

traefik would always respond with the following without ever forwarding the request to the application:

HTTP/1.1 200 OK
Access-Control-Max-Age: 0
Date: Thu, 18 Apr 2024 21:51:56 GMT
Content-Length: 0
Connection: close

When I removed the addVaryHeader: true option, requests were forwarded normally and I got the headers that my app generated.

Example with addVaryHeader: true enabled:

$ curl -sv https://my-domain.example.org \
  -X OPTIONS \
  --header 'access-control-request-method: OPTIONS' \
  --header 'origin: https://example.org'
...
> OPTIONS / HTTP/2
> Host: my-domain.example.org
> User-Agent: curl/8.7.1
> Accept: */*
> access-control-request-method: OPTIONS
> origin: https://example.org
>
* Request completely sent off
< HTTP/2 200
< access-control-max-age: 0
< content-length: 0
< date: Thu, 18 Apr 2024 21:56:38 GMT
<
* Connection #0 to host my-domain.example.org left intact

Example response from my app with the addVaryHeader removed from the configuration:

$ curl -sv https://my-domain.example.org \
  -X OPTIONS \
  --header 'access-control-request-method: OPTIONS' \
  --header 'origin: https://example.org'
...
> OPTIONS / HTTP/2
> Host: my-domain.example.org
> User-Agent: curl/8.7.1
> Accept: */*
> access-control-request-method: OPTIONS
> origin: https://example.org
>
* Request completely sent off
< HTTP/2 204
< access-control-allow-credentials: true
< access-control-allow-headers: *
< access-control-allow-methods: OPTIONS
< access-control-allow-origin: https://example.org
< date: Thu, 18 Apr 2024 22:01:53 GMT
< permissions-policy: interest-cohort=()
< referrer-policy: no-referrer-when-downgrade
< vary: Origin, Access-Control-Request-Method
...

I don't know whether this is the only situation, but replacing OPTIONS with GET (both method and header) works like it should.

maietta commented 6 months ago

Bump!

Setting the ORIGIN in my Dockerfile, re-establishes this missing header, which patches my issue for now.

I'd love to see more attention on this issue.

traefiker commented 4 months ago

Hi! I'm Træfiker :robot: the bot in charge of tidying up the issues.I have to close this one because of its lack of activity :disappointed:Feel free to re-open it or join our Community Forum.

traefik / traefik