Bouncer doesn't correctly process X-Forwarded-For headers

[thespad]

crowdsec-bouncer-traefik  | 2022-01-12T00:11:31Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.13","latency":2.450771,"user_agent":"Gatus/1.0","time":"2022-01-12T00:11:31Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-12T00:11:31Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.13","latency":2.440972,"user_agent":"Gatus/1.0","time":"2022-01-12T00:11:31Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-12T00:11:32Z DBG No decision for IP "172.20.6.1". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.20.6.1","latency":2.334519,"user_agent":"Prometheus/2.32.1","time":"2022-01-12T00:11:32Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-12T00:11:34Z DBG No decision for IP "172.70.34.108". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"44.200.29.26","latency":2.456313,"user_agent":"axios/0.21.4","time":"2022-01-12T00:11:34Z","message":"Request"}

As you can see the initial requests are internal addresses and the DBG lookup matches the subsequent request, however, the last request in the list is doing a Decision lookup for the last hop address (Cloudflare in this instance - 172.70.34.108) rather than the real address (44.200.29.26) and is allowing it even if it's banned.

In the specific instance:

crowdsec-bouncer-traefik  | 2022-01-12T00:11:54Z DBG No decision for IP "162.158.183.237". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"157.90.177.214","latency":2.464756,"user_agent":"Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)","time":"2022-01-12T00:11:54Z","message":"Request"}

+---------+----------+-------------------+-----------------------------------+--------+---------+--------------------------------+--------+--------------------+----------+
|   ID    |  SOURCE  |    SCOPE:VALUE    |              REASON               | ACTION | COUNTRY |               AS               | EVENTS |     EXPIRATION     | ALERT ID |
+---------+----------+-------------------+-----------------------------------+--------+---------+--------------------------------+--------+--------------------+----------+
| 1010386 | crowdsec | Ip:157.90.177.214 | crowdsecurity/http-bad-user-agent | ban    | US      |                             0  |      2 | 3h38m42.992170366s |     2623 |

157.90.177.214 is banned but the bouncer allows it because the lookup is performed against 162.158.183.237 (Cloudflare) instead.

[fbonalair]

Hello,

Since you are behind a load balancer, did you add its IPs as trusted in traefik? By default Traefik v2 don't trust any X-Forwarded-* headers, you have to enable it manually like this: forwarded-header. Otherwise, the IP will be the load balancer and not the client's one. Though, the webserver framework I am using for this bouncer do trust anything. I think that's why you have a difference between the debug log and the info for the bouncer.

To be sure, I have add some debug log containing request headers. Can you upgrade your container to tag v0.3.1 and activate debug with environment variable CROWDSEC_BOUNCER_LOG_LEVEL set to 0 ?

[thespad]

Yeah I've got the full CF address range in there already - everything else behind Traefik is able to pull the real source addresses.

    forwardedHeaders:
      trustedIPs:
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 127.0.0.1/32
        - 192.168.0.0/16
        - 172.16.0.0/12

Here are the logs

crowdsec-bouncer-traefik  | 2022-01-14T20:20:09Z DBG Handling forwardAuth request X-Forwarded-For=192.168.0.1 X-Real-Ip=192.168.0.1
crowdsec-bouncer-traefik  | 2022-01-14T20:20:09Z DBG No decision for IP "192.168.0.1". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.1","latency":4.474436,"user_agent":"Gatus/1.0","time":"2022-01-14T20:20:09Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:12Z DBG Handling forwardAuth request X-Forwarded-For="64.71.157.107, 172.70.211.154" X-Real-Ip="64.71.157.107, 172.70.211.154"
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"64.71.157.107","latency":2.623621,"user_agent":"Feedbin feed-id:927248 - 2 subscribers","time":"2022-01-14T20:20:12Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:12Z DBG No decision for IP "172.70.211.154". Accepting
crowdsec-bouncer-traefik  | 2022-01-14T20:20:20Z DBG Handling forwardAuth request X-Forwarded-For="34.227.193.132, 172.70.135.13" X-Real-Ip="34.227.193.132, 172.70.135.13"
crowdsec-bouncer-traefik  | 2022-01-14T20:20:20Z DBG No decision for IP "172.70.135.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"34.227.193.132","latency":2.923059,"user_agent":"Amazon Music Podcast","time":"2022-01-14T20:20:20Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG Handling forwardAuth request X-Forwarded-For=192.168.0.13 X-Real-Ip=192.168.0.13
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.13","latency":2.504812,"user_agent":"Gatus/1.0","time":"2022-01-14T20:20:21Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG Handling forwardAuth request X-Forwarded-For=192.168.0.13 X-Real-Ip=192.168.0.13
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.13","latency":2.279209,"user_agent":"Gatus/1.0","time":"2022-01-14T20:20:21Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG Handling forwardAuth request X-Forwarded-For=192.168.0.13 X-Real-Ip=192.168.0.13
crowdsec-bouncer-traefik  | 2022-01-14T20:20:21Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"192.168.0.13","latency":2.090055,"user_agent":"Gatus/1.0","time":"2022-01-14T20:20:21Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:22Z DBG Handling forwardAuth request X-Forwarded-For="52.213.77.15, 172.70.85.34" X-Real-Ip="52.213.77.15, 172.70.85.34"
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"52.213.77.15","latency":2.515989,"user_agent":"AHC/2.1","time":"2022-01-14T20:20:22Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-14T20:20:22Z DBG No decision for IP "172.70.85.34". Accepting

It seems to be pulling the wrong address out of the header. The most recent hop rather than the source address.

[fbonalair]

Are you really sure about your trusting proxies ? Cloudflare's IP 172.70.85.34 is nowhere in the trusted proxies you have send. None 172.70.X.X IP seems to be in your trusted proxies.
I have fix the debug log, Traefik actual header X-Real-IP should be correct now, add some more. Could you upgrade for the container tag v0.3.2?

[thespad]

172.70.85.34 sits within 172.64.0.0/13

I'll generate some logs for you with the new version.

[thespad]

crowdsec-bouncer-traefik  | 2022-01-16T18:24:35Z DBG Handling forwardAuth request X-Forwarded-For=192.168.0.13 X-Real-Ip=192.168.0.13
crowdsec-bouncer-traefik  | 2022-01-16T18:24:35Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=192.168.0.13
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":2.193192,"user_agent":"Gatus/1.0","time":"2022-01-16T18:24:35Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-16T18:24:35Z DBG No decision for IP "192.168.0.13". Accepting
crowdsec-bouncer-traefik  | 2022-01-16T18:24:44Z DBG Handling forwardAuth request X-Forwarded-For="192.0.101.226, 172.70.175.35" X-Real-Ip=172.70.175.35
crowdsec-bouncer-traefik  | 2022-01-16T18:24:44Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=172.70.175.35
crowdsec-bouncer-traefik  | 2022-01-16T18:24:44Z DBG No decision for IP "172.70.175.35". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":2.511431,"user_agent":"jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)","time":"2022-01-16T18:24:44Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-16T18:25:09Z DBG Handling forwardAuth request X-Forwarded-For="35.84.190.214, 108.162.246.25" X-Real-Ip=108.162.246.25
crowdsec-bouncer-traefik  | 2022-01-16T18:25:09Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=108.162.246.25
crowdsec-bouncer-traefik  | 2022-01-16T18:25:09Z DBG No decision for IP "108.162.246.25". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":3.063167,"user_agent":"Amazon Music Podcast","time":"2022-01-16T18:25:09Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-16T18:25:10Z DBG Handling forwardAuth request X-Forwarded-For="52.213.26.200, 172.70.91.111" X-Real-Ip=172.70.91.111
crowdsec-bouncer-traefik  | 2022-01-16T18:25:10Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=172.70.91.111
crowdsec-bouncer-traefik  | 2022-01-16T18:25:10Z DBG No decision for IP "172.70.91.111". Accepting
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":4.40267,"user_agent":"AHC/2.1","time":"2022-01-16T18:25:10Z","message":"Request"}

Looks like it's still querying the CF IP:

crowdsec-bouncer-traefik  | 2022-01-16T18:25:10Z DBG Handling forwardAuth request X-Forwarded-For="52.213.26.200, 172.70.91.111" X-Real-Ip=172.70.91.111
crowdsec-bouncer-traefik  | 2022-01-16T18:25:10Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=172.70.91.111

[thespad]

Go is very much not my strong suit but I've been trying to get a handle on why the behaviour isn't as expected and I think I've got a sense of it now.

So the X-Forwarded-For header is the defacto standard for source (and intermediate) IP address recording. Proxies and other intercepting services are permitted to append to the X-Forwarded-For header so that you have a "paper trail" of the hops the request has passed through, but the first value should be the originating IP. X-Real-IP doesn't support appending and while it may show the originating IP there's no way to know it hasn't been overwritten along the way (technically this is also true with X-Forwarded-For but it at least allows appending). Consequently, it's much more reliable to use the X-Forwarded-For header whenever there's a possibility of more than one proxy between the origin and the destination server.

For example, in this request:

crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG Handling forwardAuth request X-Forwarded-For="34.220.8.237, 162.158.107.233" X-Real-Ip=162.158.107.233
crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=162.158.107.233
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":2.872559,"user_agent":"Amazon Music Podcast","time":"2022-01-25T11:45:06Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG No decision for IP "162.158.107.233". Accepting

Here, 162.158.107.233 is a Cloudflare address, showing up as both the whole X-Real-IP header and the last entry in X-Forwarded-For. 34.220.8.237 is the originating IP, which also shows up in the X-Forwarded-For header as the first address.

Traefik's own logging shows:

34.220.8.237 - - [25/Jan/2022:11:45:06 +0000] "GET /feed/dungeons/ HTTP/2.0" 200 52493 "-" "Amazon Music Podcast" 46091 "https-tsp-all@docker" "http://172.18.0.23:80" 138ms

Your current code is using the X-Real-IP header to do its auth checks, which is only going to show the last hop before Traefik, which in my case is Cloudflare's edge.

I think it should be as straightforward as using the X-Forwarded-For header and extracting the first address from the list.

[pewter77]

I've opened a pull request for this, happy to talk through it or other concerns. I've tested it working for me (specifically sitting behind Cloudflare CDN) but I don't have much golang experience so there might be formatting or other issues with it.

[jonwilliams84]

+1 for this behaviour behind cloudflare CDN.

[thespad]

In the interim I've forked the repo here https://github.com/thespad/traefik-crowdsec-bouncer and merged @pewter77 's PR.

[jonwilliams84]

In the interim I've forked the repo here https://github.com/thespad/traefik-crowdsec-bouncer and merged @pewter77 's PR.

I have updated my stack to use your image. Am I correct in thinking that I need to pass the Cloudflare CIDRs into the TRUSTED_PROXIES env?

TRUSTED_PROXIES: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"

I have tried this and also tried it set as default, but it's still not blocking clients that come through from CDN.

Ta

[thespad]

Should just be the Traefik CIDRs as that's the immediate upstream source (Traefik should already trust the Cloudflare addresses).

If you set CROWDSEC_BOUNCER_LOG_LEVEL=0 it'll give you the full debug output in the console and you can see the decision process.

It's worth noting that I am aware of an issue with the Traefik scenarios in Crowdsec if you're using JSON logging, it doesn't pick up the true client address due to different use of fields.

[pewter77]

Quite honestly you don't need to set the trusted proxy setting unless you're setting it up on a different machine or something. Traefik will handle which are trusted before it hits this one. Keeping it blank will set it to trust all forwarding traffic. You could set it to trust docker networks if you really want to. The setting is about trusting the proxy in front of the bouncer really not necessarily cloudflare but you'd need to do testing.

You do need to set cloudflare CDN as trusted in the traefik config though.

[fbonalair]

So the X-Forwarded-For header is the defacto standard for source (and intermediate) IP address recording. Proxies and other intercepting services are permitted to append to the X-Forwarded-For header so that you have a "paper trail" of the hops the request has passed through, but the first value should be the originating IP. X-Real-IP doesn't support appending and while it may show the originating IP there's no way to know it hasn't been overwritten along the way (technically this is also true with X-Forwarded-For but it at least allows appending). Consequently, it's much more reliable to use the X-Forwarded-For header whenever there's a possibility of more than one proxy between the origin and the destination server.

That's what I have concluded too. I just did not have time to fix it until now.

I've opened a pull request for this, happy to talk through it or other concerns. I've tested it working for me (specifically sitting behind Cloudflare CDN) but I don't have much golang experience so there might be formatting or other issues with it.

Thanks for your work! I will merge it, it's an additional feature for the project. Don't worry about golang, I'm a noob myself, I have used this project to work on the language actually ^^.

Should just be the Traefik CIDRs as that's the immediate upstream source (Traefik should already trust the Cloudflare addresses).

Exactly, by default the Web framework I'm using is trusting everything by default. For this service and its scope, it should be fine.

This trusted proxy issue should be fixed from version 0.3.4 of the container.

[aleksandarmomic]

https://github.com/fbonalair/traefik-crowdsec-bouncer/pull/14 fixed my problems. Thanks and great job @pewter77!

[vherrlein]

I confirm too, that’s working fine now, even behind 2 reverse proxies .

[Xoffio]

Hi guys! I think I have similar or the same problem. I tried the latest version from both repos.

I am running crowdsec and this bouncer on a k3s cluster with traefik as ingress. When I access the website from my browser and look at the bouncer's logs, this is what I get:

2022-02-25T04:17:52Z DBG Handling forwardAuth request ClientIP=10.42.0.123 X-Forwarded-For=10.42.1.0 X-Real-Ip=10.42.1.0
2022-02-25T04:17:52Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec-service.crowdsec:8080/v1/decisions?type=ban&ip=10.42.0.123
2022-02-25T04:17:52Z DBG No decision for IP "10.42.0.123". Accepting
{"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"10.42.0.123","latency":36.3995,"user_agent":"Mozilla/5.0  (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101  Firefox/98.0","time":"2022-02-25T04:17:52Z","message":"Request"}

when I check where that ip comes from I get that is from the traefik pod

> kubectl get pods -o wide --all-namespaces | grep "123"
kube-system          traefik-ffd48d555-wfpg4         1/1     Running       0                9h      10.42.0.123     main-node01

Am I missing something or this bug still present?

Thanks

[vherrlein]

Hi guys! I think I have similar or the same problem. I tried the latest version from both repos.

I am running crowdsec and this bouncer on a k3s cluster with traefik as ingress. When I access the website from my browser and look at the bouncer's logs, this is what I get:

2022-02-25T04:17:52Z DBG Handling forwardAuth request ClientIP=10.42.0.123 X-Forwarded-For=10.42.1.0 X-Real-Ip=10.42.1.0
2022-02-25T04:17:52Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec-service.crowdsec:8080/v1/decisions?type=ban&ip=10.42.0.123
2022-02-25T04:17:52Z DBG No decision for IP "10.42.0.123". Accepting
{"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"10.42.0.123","latency":36.3995,"user_agent":"Mozilla/5.0  (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101  Firefox/98.0","time":"2022-02-25T04:17:52Z","message":"Request"}

when I check where that ip comes from I get that is from the traefik pod

> kubectl get pods -o wide --all-namespaces | grep "123"
kube-system          traefik-ffd48d555-wfpg4         1/1     Running       0                9h      10.42.0.123     main-node01

Am I missing something or this bug still present?

Thanks

It’s mainly an issue with your setup of traefik entry point. Within Kubernetes, Traefik Agents act as Network Load Balancers, so you must configure the EntryPoint Option “Forwarded Headers”.

https://doc.traefik.io/traefik/routing/entrypoints/#forwarded-headers

In addition, if you have another reverse proxy in front of Traefik, you must:

1. Ensure your front Reverse Proxy uses Proxy Protocol (Automatic with CloudFlare, setting up send-proxy-v2 within backend service of HAProxy)
2. Configure your Traefik entrypoint with ProxyProtocol

https://doc.traefik.io/traefik/routing/entrypoints/#proxyprotocol

Last but not least, make sure your Traefik middleware “FowardAuth” uses the option “trustForwardHeaders” https://doc.traefik.io/traefik/middlewares/http/forwardauth/#trustforwardheader

Sample Setup

**values.yaml** ```yaml additionalArguments: - --providers.file.filename=/data/traefik-config.yaml - --entrypoints.websecure.http.middlewares=hardening@file - --entrypoints.websecure.http.tls=true - --entrypoints.websecure.http.tls.certresolver=leresolver - --entrypoints.websecure.http.tls.domains[0].main=yourdomain.com - --entrypoints.websecure.http.tls.domains[0].sans=*.yourdomain.com - --entrypoints.websecure.proxyProtocol.insecure=true - --entrypoints.websecure.forwardedHeaders.insecure - --certificatesresolvers.leresolver.acme.dnschallenge.provider=cloudflare - --certificatesresolvers.leresolver.acme.dnschallenge.resolvers=8.8.8.8 - --certificatesresolvers.leresolver.acme.email=cloudflare-email-account@domain.com - --certificatesresolvers.leresolver.acme.storage=/certs/acme.json ports: web: redirectTo: websecure dnsudp: port: 5553 hostPort: 5553 expose: true exposedPort: 53 protocol: UDP dnstls: port: 5853 hostPort: 5853 expose: true exposedPort: 853 protocol: TCP metrics: port: 9100 expose: true exposedPort: 9205 protocol: TCP env: - name: CF_DNS_API_TOKEN valueFrom: secretKeyRef: key: apiKey name: cloudflare-api-credentials - name: CLOUDFLARE_PROPAGATION_TIMEOUT value: "300" ingressRoute: dashboard: enabled: false providers: kubernetesCRD: allowCrossNamespace: true persistence: enabled: true path: /certs size: 128Mi volumes: - mountPath: /data name: traefik-config type: configMap pilot: enabled: true token: PILOT-TOKEN experimental: plugins: enabled: true logs: general: level: ERROR access: enabled: true ``` **Traefik Config Map & Cloudflare Secret** ```yaml --- apiVersion: v1 kind: Secret metadata: name: cloudflare-api-credentials namespace: kube-system type: Opaque stringData: email: cloudflare-email-account@domain.com apiKey: cloudflare-api-token --- apiVersion: v1 kind: ConfigMap metadata: name: traefik-config namespace: kube-system data: traefik-config.yaml: | http: middlewares: hardening: chain: middlewares: - request-limits@file - secure-headers@file - crowdsec-bouncer@file request-limits: ratelimit: average: 100 burst: 50 secure-headers: headers: browserXssFilter: true contentSecurityPolicy: "default-src 'self'; connect-src * 'self'; img-src * 'self' data: https:; object-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https:; style-src 'self' 'unsafe-inline' https:; frame-src 'self' https:; child-src 'self' https:; font-src 'self' data:" contentTypeNosniff: true customresponseheaders: Permissions-Policy: "geolocation=(self), microphone=(), camera=(), fullscreen=(self)" Server: "" X-Powered-By: "" permissionsPolicy: connect-src * 'self'; "vibrate 'self'; geolocation 'self'; midi 'self'; notifications 'self'; push 'self'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; speaker 'none'; vibrate 'self'; fullscreen 'self'; frame-src 'self' https:; child-src 'self' https:; font-src 'self' data:" forceSTSHeader: true frameDeny: false referrerPolicy: same-origin stsIncludeSubdomains: true stsSeconds: 31536000 crowdsec-bouncer: forwardauth: address: http://crowdsec-traefik-bouncer.crowdsec:8080/api/v1/forwardAuth trustForwardHeader: true ```

[fbonalair]

Hi guys! I think I have similar or the same problem. I tried the latest version from both repos.

I am running crowdsec and this bouncer on a k3s cluster with traefik as ingress. When I access the website from my browser and look at the bouncer's logs, this is what I get:

2022-02-25T04:17:52Z DBG Handling forwardAuth request ClientIP=10.42.0.123 X-Forwarded-For=10.42.1.0 X-Real-Ip=10.42.1.0
2022-02-25T04:17:52Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec-service.crowdsec:8080/v1/decisions?type=ban&ip=10.42.0.123
2022-02-25T04:17:52Z DBG No decision for IP "10.42.0.123". Accepting
{"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"10.42.0.123","latency":36.3995,"user_agent":"Mozilla/5.0  (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101  Firefox/98.0","time":"2022-02-25T04:17:52Z","message":"Request"}

when I check where that ip comes from I get that is from the traefik pod

> kubectl get pods -o wide --all-namespaces | grep "123"
kube-system          traefik-ffd48d555-wfpg4         1/1     Running       0                9h      10.42.0.123     main-node01

Am I missing something or this bug still present?

Thanks

Moreover what @vherrlein said, there's a whole discussion on github about this problem. For my cluster, this answer was my salvation : https://github.com/k3s-io/k3s/discussions/2997#discussioncomment-569799

[Xoffio]

Hi. I have been working on this problem the last 3 days. First of all, thank you guys for taking the time to help me!

@vherrlein, at first I tried adding the forwardedHeaders.insecure CLI options into my HelmChartConfig file (traefik-config.yaml). Like here

additionalArguments:
      - --entrypoints.websecure.proxyProtocol.insecure=true
      - --entrypoints.websecure.forwardedHeaders.insecure

Also, my Traefik middleware “FowardAuth” uses the option “trustForwardHeaders.”

For some reason this didn't work. Or at least not just by adding those extra flags. Then I read what @fbonalair posted. At the beginning I didn't know how to do it until I found this.

So I also added this to my file:

service:
      spec:
        externalTrafficPolicy: Local

[vherrlein]

@SpaceComet what is your CNI ? If it’s the default one with K3s, you should have Flannel which would be the culprit. Flannel CNI create an overlay network using by default VXLAN protocol.

In my setup, I’m using Calico as CNI for managing Kubernetes networks. The main advantage of Calico, it uses BGP Protocol for routing, so no additional extra net layer wrapped, meaning an higher MTU :)

The trick with “externalTrafficPolicy: Local” works, but only if you don’t need HA with your pods.

If you want to try Calico, that’s pretty straightforward and very well documented for K3s: https://projectcalico.docs.tigera.io/getting-started/kubernetes/k3s/

To compare Flannel vs Calico : https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/

K3s & Calico Setup sample

1. Prerequisites, setup all K3s master nodes with --cluster-cidr=172.16.0.0/12

On each MASTER node, k3s.service:

[Service]
MountFlags=shared
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--flannel-backend=none' \
        '--disable-network-policy' \
        '--cluster-cidr=172.16.0.0/12' \
        --disable traefik

2. Install Calico Operator

kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml

3. Install custom Calico manifest for the target CDIR

   kubectl create -f calico-custom-resources.yaml

calico-custom-resources.yaml

# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/v3.22/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 172.16.0.0/12
      encapsulation: IPIP
      natOutgoing: Enabled
      nodeSelector: all()

---

# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/v3.22/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer 
metadata: 
  name: default 
spec: {}

[Xoffio]

Hi @vherrlein! Thanks for the valuable info! I was actually concern about not having HA. So much that I redoo my tmp cluster to see what can I change. I actually don't now. I guess the default one. What I noticed with my current setup is that traefik only deploy one pod and 3 others as load balancer but if the node where traefik lives dies then my cluster go down :(

> kubectl get pods -n kube-system | grep traefik
helm-install-traefik-crd--1-n5mhh         0/1     Completed   0             25h
svclb-traefik-jbrbh                       2/2     Running     0             25h
svclb-traefik-4m9rv                       2/2     Running     0             25h
svclb-traefik-pwnbw                       2/2     Running     0             25h
helm-install-traefik--1-9spfw             0/1     Completed   1             37m
traefik-7b9b956bb9-2n6x9                  1/1     Running     0             37m # this one

So if I install calico this should be solved right ?

[vherrlein]

I was actually concern about not having HA.

Mainly, if you don't have a Traefik Pod Instance per node, using "externaltrafficpolicy: Local" on the service will give you some wired routing situations (black hole, traffic routed on single node,...)

Well detailed explanations: https://technotes.adelerhof.eu/containers/kubernetes/externaltrafficpolicy/ Official Kubernetes Warnings: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

What I noticed with my current setup is that traefik only deply one pod and 3 others as load balancer but if the node where traefik lives dies then my cluster go down

In that case, as you are using Helm, that's pretty easy, play with the "replicas" option into your helm chart values file.

However, according to Traefik documentations, certificate management in HA way is not implemented anymore (that feature was present in v1.x), only in the enterprise edition. (cf. https://doc.traefik.io/traefik-enterprise/features/#compare)

So, if you want to do it, make sure your SSL certificates are managed externally of Traefik, example by using "cert-manager". (eg. https://www.scaleway.com/en/docs/tutorials/traefik-v2-cert-manager/)

So if I install calico this should be solved right ?

By using Calico, you will have a better control of your ingress network and less pains with reverse proxies source ips. Then it should be solved if all points discussed previously are in place.

[yfhyou]

Hate to jump into this already long thread, but I'm struggling with this issue. I'm using cloudflare and have set the trusted IPs for cloudflare in my traefik entry point. However, the IP that is being inspected by the bouncer is still the CF proxy one:

traefikbouncer    | 2022-03-04T14:46:16Z DBG Handling forwardAuth request ClientIP=108.162.237.158 RemoteAddr=172.19.0.20:56104 X-Forwarded-For=108.162.237.158 X-Real-Ip=108.162.237.158
traefikbouncer    | 2022-03-04T14:46:16Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=108.162.237.158
traefikbouncer    | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"108.162.237.158","latency":1.401838,"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0","time":"2022-03-04T14:46:16Z","message":"Request"}
traefikbouncer    | 2022-03-04T14:46:16Z DBG No decision for IP "108.162.237.158". Accepting

Meanwhile the whoami container shows:

Cf-Connecting-Ip: {my.real.ip.here}
...
X-Forwarded-For: {my.real.ip.here}, 108.162.237.158
X-Forwarded-Host: {whoami.my.website}
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: a53ae7465bc8
X-Real-Ip: 108.162.237.158

Could the Cf-Connecting-Ip header be used?

I've tried using the traefik-real-ip plugin, and while it does fix the X-Real-Ip header:

X-Forwarded-For: {my.real.ip.here}, 108.162.238.135
X-Forwarded-Host: {whoami.my.website}
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: efa076724706
X-Real-Ip: {my.real.ip.here}

This bouncer still inspects the wrong IP:

traefikbouncer    | 2022-03-04T14:52:51Z DBG Handling forwardAuth request ClientIP=108.162.238.135 RemoteAddr=172.19.0.20:56106 X-Forwarded-For=108.162.238.135 X-Real-Ip={my.real.ip.here}
traefikbouncer    | 2022-03-04T14:52:51Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=108.162.238.135
traefikbouncer    | 2022-03-04T14:52:51Z DBG No decision for IP "108.162.238.135". Accepting

I've tried adding the cloudflare IPs to the TRUSTED_PROXIES env variable for my bouncer, but then it returned the traefik docker address (172.20.0.20). I hope I'm over complicating things and missing something simple :)

[vherrlein]

@youngt2, my whoami container displays the same results as you, which is normal for the X-ForwardedFor Header. And as usual the X-ClientIP doesn’t work properly.

check your Traefik settings, like discussed here

[yfhyou]

Thank you @vherrlein!

@youngt2, my whoami container displays the same results as you, which is normal for the X-ForwardedFor Header. And as usual the X-ClientIP doesn’t work properly.

check your Traefik settings, like discussed here

Seems I was missing the trustForwardHeader: true for my bouncer middleware. I definitely read that part, but didn't connect the dots... I don't understand the ProxyProtocol though. It seems its not needed? Is it in addition to the forwardedHeaders, or something totally different? I can only find it with cloudflare 'spectrum' not with a standard free plan.

[Xoffio]

@vherrlein Thank you so much man! The "replicas" flag solved that and I already had setup cert-manager. Right now I am installing Calico. What is the name of the file that starts like this:

[Service]
MountFlags=shared
Type=notify
EnvironmentFile
...

[vherrlein]

@youngt2, don’t worry about the ProxyProtocol from Cloudflare it’s automated. I’m using it too with a free account for my lab without any additional settings to make it works.

Only if you have a LoadBalancer or a Reverse Proxy in between your Kubernetes infrastructure and your internet router.

@SpaceComet, it’s the “k3s.service” file describing the service. If your OS uses systemd it should be located here: /etc/systemd/system/k3s.service