[BUG] Can't Get Real IP, getting Docker Bridge Gateway IP instead

xlionjuan commented 1 year ago

Describe the bug 🐛 Logs from Traefik:

DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:06 New initialized mode:live
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:49 ServeHTTP ip:172.26.0.1 isTrusted:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:49 cache:GetDecision ip:172.26.0.1
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:49 ServeHTTP:getDecision ip:172.26.0.1 isBanned:false cache:miss
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:49 ServeHTTP:handleNoStreamCache ip:172.26.0.1 isBanned:true crowdsecQuery url:http://crowdsec:8080/v1/decisions?ip=172.26.0.1&banned=true, statusCode:403
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:28:14 ServeHTTP ip:172.26.0.1 isTrusted:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:28:14 cache:GetDecision ip:172.26.0.1
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:28:14 ServeHTTP:getDecision ip:172.26.0.1 isBanned:false cache:miss
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:28:14 ServeHTTP:handleNoStreamCache ip:172.26.0.1 isBanned:true crowdsecQuery url:http://crowdsec:8080/v1/decisions?ip=172.26.0.1&banned=true, statusCode:403

log from crowdsec:

time="16-10-2023 14:41:24" level=error msg="while fetching bouncer info: select bouncer: ent: bouncer not found: unable to query" ip=172.26.0.3
time="16-10-2023 14:41:24" level=info msg="172.26.0.3 - [Mon, 16 Oct 2023 14:41:24 UTC] \"GET /v1/decisions?ip=172.26.0.1&banned=true HTTP/1.1 403 630.628µs \"Go-http-client/1.1\" \""
time="16-10-2023 14:41:24" level=error msg="UnmarshalJSON : invalid character '.' after top-level value" line="172.26.0.1 - - [16/Oct/2023:14:41:24 +0000] \"GET / HTTP/2.0\" 403 742 \"-\" \"-\" 41 \"router-{deleted}@file\" \"http://error_pages_serv:80\" 6ms"
time="16-10-2023 14:41:24" level=warning msg="failed to run filter : invalid character '.' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
time="16-10-2023 14:41:25" level=error msg="while fetching bouncer info: select bouncer: ent: bouncer not found: unable to query" ip=172.26.0.3
time="16-10-2023 14:41:25" level=info msg="172.26.0.3 - [Mon, 16 Oct 2023 14:41:25 UTC] \"GET /v1/decisions?ip=172.26.0.1&banned=true HTTP/1.1 403 674.255µs \"Go-http-client/1.1\" \""
time="16-10-2023 14:41:25" level=error msg="UnmarshalJSON : invalid character '.' after top-level value" line="172.26.0.1 - - [16/Oct/2023:14:41:25 +0000] \"GET / HTTP/2.0\" 403 742 \"-\" \"-\" 42 \"router-{deleted}@file\" \"http://error_pages_serv:80\" 4ms"
time="16-10-2023 14:41:25" level=warning msg="failed to run filter : invalid character '.' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
time="16-10-2023 14:41:35" level=info msg="127.0.0.1 - [Mon, 16 Oct 2023 14:41:35 UTC] \"GET /v1/heartbeat HTTP/1.1 200 1.289397187s \"crowdsec/v1.5.4-e4dcdd25728b914823525f1efabf18d5c454902b\" \""
time="16-10-2023 14:41:42" level=error msg="while fetching bouncer info: select bouncer: ent: bouncer not found: unable to query" ip=172.26.0.3
time="16-10-2023 14:41:42" level=info msg="172.26.0.3 - [Mon, 16 Oct 2023 14:41:42 UTC] \"GET /v1/decisions?ip=172.26.0.1&banned=true HTTP/1.1 403 705.6µs \"Go-http-client/1.1\" \""
time="16-10-2023 14:41:42" level=error msg="UnmarshalJSON : invalid character '.' after top-level value" line="172.26.0.1 - - [16/Oct/2023:14:41:42 +0000] \"GET / HTTP/2.0\" 403 742 \"-\" \"-\" 44 \"router-{deleted}@file\" \"http://error_pages_serv:80\" 5ms"
time="16-10-2023 14:41:42" level=warning msg="failed to run filter : invalid character '.' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-sun name=child-crowdsecurity/traefik-logs stage=s01-parse
time="16-10-2023 14:41:47" level=error msg="while fetching bouncer info: select bouncer: ent: bouncer not found: unable to query" ip=172.26.0.3
time="16-10-2023 14:41:47" level=info msg="172.26.0.3 - [Mon, 16 Oct 2023 14:41:47 UTC] \"GET /v1/decisions?ip=172.26.0.1&banned=true HTTP/1.1 403 637.982µs \"Go-http-client/1.1\" \""

-10-2023 14:41:47" level=error msg="UnmarshalJSON : invalid character '.' after top-level value" line="172.26.0.1 - - [16/Oct/2023:14:41:47 +0000] \"GET / HTTP/2.0\" 403 742 \"-\" \"-\" 45 \"router-{deleted}@file\" \"http://error_pages_serv:80\" 12ms"
time="16-10-2023 14:41:47" level=warning msg="failed to run filter : invalid character '.' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-sun name=child-crowdsecurity/traefik-logs stage=s01-parse

Applications behind Traefik can get Real IP, whoani shows it can able get X-Forwarded-For and X-Real-Ip, accesslog can get real IP also. cscli decisions add --ip 1.2.3.4 --duration 24h --reason "web bruteforce" this command from Crowdsec can also ban my IP

Expected behavior 👀 Get Real IP

Context 🔎 middlewares config:

http:
  middlewares:
    crowdsec:
      plugin:
        crowdsec-bouncer-traefik-plugin:
          enabled: true
          logLevel: DEBUG
          updateIntervalSeconds: 60
          defaultDecisionSeconds: 60
          httpTimeoutSeconds: 10
          crowdsecMode: live
          crowdsecLapiKey: ######
          #crowdsecLapiKeyFile: /etc/traefik/cs-privateKey-foo
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiScheme: http
          crowdsecLapiTLSInsecureVerify: false
          crowdsecCapiMachineId: login
          crowdsecCapiPassword: password
          crowdsecCapiScenarios:
            - crowdsecurity/http-path-traversal-probing
            - crowdsecurity/http-xss-probing
            - crowdsecurity/http-generic-bf
          forwardedHeadersTrustedIPs: 
            - 172.26.0.1 # <-- I've tried this or not tried this, but didn't solved.

          clientTrustedIPs: 
            - 192.168.2.0/24
          forwardedHeadersCustomName: X-Real-IP #  <-- I've Tried ``X-Forwarded-For``, ``X-Real-IP`` or without this value.
          redisCacheEnabled: false
          redisCacheHost: "redis:6379"
          redisCachePassword: password
          redisCacheDatabase: "5"

Version (please complete the following information):

OS: Ubuntu Core 22
Traefik version: 2.10.5
Plugin version: 1.1.16
Redis ? : N/A

To Reproduce Steps to reproduce the behavior:

mathieuHa commented 1 year ago

Hello @xlionjuan

I see that you are using live mode. With this you don't have to set thoses variables (below) for CAPI (central API) because it means you are running a LAPI crowdsec container which does processing of your logs.

          crowdsecCapiMachineId: login
          crowdsecCapiPassword: password
          crowdsecCapiScenarios:
            - crowdsecurity/http-path-traversal-probing
            - crowdsecurity/http-xss-probing
            - crowdsecurity/http-generic-bf

You do have to configure crowdsec to enable the scenario you want for instance like this:

  crowdsec:
    image: crowdsecurity/crowdsec:v1.5.3
    container_name: "crowdsec"
    restart: unless-stopped
    environment:
      COLLECTIONS: crowdsecurity/traefik
      CUSTOM_HOSTNAME: crowdsec
      # We need to register one api key per service we will use
      BOUNCER_KEY_TRAEFIK: FIXME-LAPI-KEY-1=
    volumes:
      - ./acquis.yaml:/etc/crowdsec/acquis.yaml:ro
      - logs:/var/log/traefik:ro
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
    labels:
      - "traefik.enable=false"

From this log and the last part statusCode:403, I think your connection to the crowdsec LAPI is not authenticated correctly.

DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/16 22:27:49 ServeHTTP:handleNoStreamCache ip:172.26.0.1 isBanned:true crowdsecQuery url:http://crowdsec:8080/v1/decisions?ip=172.26.0.1&banned=true, statusCode:403

Can you recrate a LAPI key and add it in the bouncer configuration ?

mathieuHa commented 1 year ago

172.17.0.0/16 is usually the network range for docker.

if you see in the logs

ServeHTTP ip:172.26.0.1 isTrusted:false

It probably means the X-forwarded-for from the header is not picked up correctly

mathieuHa commented 1 year ago

From Crowdsec logs:

time="16-10-2023 14:41:24" level=warning msg="failed to run filter : invalid character '.' after top-level value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=restless-sun name=child-crowdsecurity/traefik-logs stage=s01-parse

It seams to receive some event but is not able to parse them.
Can you share your logging configuration of traefik ?

Did you add the parser for traefik ? Installed with this env var for instance: COLLECTIONS: crowdsecurity/traefik

mathieuHa commented 1 year ago

If you could provide your docker-compose (with the strict minimum) it could be helpful. In the meantime you may start from this one which is supposed to work well and test it with something like Whoami.

xlionjuan commented 1 year ago

Hello @xlionjuan

I see that you are using live mode. With this you don't have to set thoses variables (below) for CAPI (central API) because it means you are running a LAPI crowdsec container which does processing of your logs.
          crowdsecCapiMachineId: login
          crowdsecCapiPassword: password
          crowdsecCapiScenarios:
            - crowdsecurity/http-path-traversal-probing
            - crowdsecurity/http-xss-probing
            - crowdsecurity/http-generic-bf
@mathieuHa Thank you, I only delete these lines and recreate LAPI key, and it can get the client's real IP again!

xlionjuan commented 1 year ago

I didn't see "Mark as answer" button, maybe you can enable it?

maxlerebourg / crowdsec-bouncer-traefik-plugin

[BUG] Can't Get Real IP, getting Docker Bridge Gateway IP instead #119