search

maxlerebourg / crowdsec-bouncer-traefik-plugin

Traefik plugin for Crowdsec - WAF and IP protection
Apache License 2.0
273 stars 13 forks source link

clientTrustedIPs seems to block every IP except it's parameters #41

Closed rwjack closed 2 years ago

rwjack commented 2 years ago

Hey @mathieuHa, just wanted to let you know that I'm having some issues accessing services from the outside.

Now everyone gets 403 responses. Disabling the bouncer from the traefik static config fixes the issue, meaning the plugin is definitely the culprit.

Consider that everything on localnet is 172.16.1.1/24

I've set:

clientTrustedIPs:
  - 172.16.1.1/24

And this works as expected, these clients are not rate limited and don't go through crowdsec.

Though when adding - 0.0.0.0/0, I have access all services externally.

It seems now that only clientTrustedIPs are allowed to connect, regardless of what crowdsec says.

Any thoughts?

I'm using Traefik 2.8.4, and the latest commit of this plugin (deployed in the plugins-local dir). I don't use redis for now. I'll look into debug mode a bit later, if it's even required at all, though this seems like some sort of funny semantic issue :)

rwjack commented 2 years ago

Config which returns 403 to everyone, excluding 172.16.1.1/24:

  middlewares:
    crowdsec:
      plugin:
        bouncer:
          enabled: true
          updateIntervalSeconds: 60
          crowdsecMode: stream
          crowdsecLapiKey: foobar
          crowdsecLapiHost: host.fqdn:1234
          crowdsecLapiScheme: http
          clientTrustedIPs:
            - 172.16.1.1/24

Config which returns 20* to everyone:

          clientTrustedIPs:
            - 0.0.0.0/0

Config which returns 403 to everyone:

          #clientTrustedIPs:
          #  - 0.0.0.0/0
maxlerebourg commented 2 years ago

Cool, thanks for more advanced configuration, maybe you can use logLevel: DEBUG option to get information.

The first thing that comes to mind is that your crowdsec instance is unavailable or cannot be reached by the plugin.

mathieuHa commented 2 years ago

Hi @rwjack

We usually use the following container to debug things with Traefik: https://hub.docker.com/r/traefik/whoami

When you access it, and it gives you a 200 status, it will show you what are the IPs are seen by the end service.

For instance:

Hostname: 78f9564adb82
IP: 127.0.0.1
IP: 192.168.48.3
RemoteAddr: 192.168.48.2:40642
GET /foo HTTP/1.1
Host: 10.0.20.40
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Dnt: 1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.0.10.5
X-Forwarded-Host: 10.0.20.40
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 98e7afca71de
X-Real-Ip: 10.0.10.5

Here 10.0.10.5 is my true IP as seen with traefik/whoami container 10.0.20.40 si my destination

You used clientTrustedIPs:[172.16.1.1/24] It looks to me like the docker internal network, is that the case ?

I'm using Traefik 2.8.4, and the latest commit of this plugin (deployed in the plugins-local dir).

Do you mean you deployed the plugin with the sourcecode of the repository binded in the container ?

    command:
      - "--log.level=DEBUG"
      - "--accesslog"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"

      - "--experimental.localplugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - logs-local:/var/log/traefik
      - ./:/plugins-local/src/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin

I recommand you to try and deploy it if that's the case with the release way:

    command:
...
      - "--experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.bouncer.version=v1.1.4"

It would be much easier to fix this issue if you could try and run the plugin in debug mode:

In the label of your service add the following;

      - "traefik.http.middlewares.CROWDSEC-FIXME-INSTANCENAME.plugin.bouncer.loglevel=DEBUG"

Also we could use the logs from crowdsec container.

Please remove any sensitive information before posting but keep a consistent naming about what IP replaced in the logs is supposed to be (client, container, proxy?)

Thanks you by advance

rwjack commented 2 years ago

_crowdsec_logs.txt

Hey, I attached the last 5000 lines of the Crowdsec container, and replaced my Traefik VM IP with traefik.vm.ip.addr.

The Crowdsec container seems reachable from the Traefik VM, and do note that I'm running Traefik on Debian, not as a container.

Also, I did not really use clientTrustedIPs:[172.16.1.1/24], it was just an example, but maybe better to use the A class as an example (which I actually use), so 10.0.0.0/24.

Do you mean you deployed the plugin with the sourcecode of the repository binded in the container ?

Technically, yes, but not bound in the container since I'm running traefik on Debian with systemd. This entire git repo of this plugin is located in: /opt/traefik/plugins-local/src/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/

traefik.yml:

experimental:
  #plugins:
  localPlugins:
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      #version: "v1.1.0"

I recommand you to try and deploy it if that's the case with the release way:

I couldn't get that working, because this VM has blocked all outbound internet access unless it's permitted URLs through a proxy, so traefik is having problems talking to github. That's why I manually deployed the plugin in plugins-local, as stated in Traefik documentation.

It would be much easier to fix this issue if you could try and run the plugin in debug mode:

I have added the following in the dynamic conf:

http:
  middlewares:
    crowdsec:
      plugin:
        bouncer:
        ...
        logLevel: DEBUG
        ...

Do I also need to change this in the static conf:

log:
  filePath: /var/log/traefik/traefik.log
  level: INFO # -> DEBUG?

Because the logs don't show much:

$ systemctl restart traefik && tail -f /var/log/traefik/traefik.log
time="2022-11-26T13:57:53+01:00" level=error msg="Error while Peeking first byte: read tcp traefik.vm.ip.addr:443->external.ip:64950: read: connection timed out"
time="2022-11-26T14:08:49+01:00" level=error msg="Error while Peeking first byte: read tcp traefik.vm.ip.addr:443->external.ip:22724: read: connection timed out"
time="2022-11-26T14:13:45+01:00" level=info msg="I have to go..."
time="2022-11-26T14:13:45+01:00" level=info msg="Stopping server gracefully"
time="2022-11-26T14:13:45+01:00" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2022-11-26T14:13:45+01:00" level=error msg="Error while starting server: accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2022-11-26T14:13:45+01:00" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2022-11-26T14:13:45+01:00" level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2022-11-26T14:13:55+01:00" level=info msg="Server stopped"
time="2022-11-26T14:13:55+01:00" level=info msg="Shutting down"
time="2022-11-26T14:13:55+01:00" level=info msg="Traefik version 2.8.4 built on 2022-09-02T14:42:59Z"
time="2022-11-26T14:13:55+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2022-11-26T14:13:55+01:00" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
time="2022-11-26T14:13:55+01:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2022-11-26T14:13:55+01:00" level=info msg="Starting provider *file.Provider"
time="2022-11-26T14:13:55+01:00" level=info msg="Starting provider *traefik.Provider"
time="2022-11-26T14:13:55+01:00" level=info msg="Starting provider *acme.Provider"
time="2022-11-26T14:13:55+01:00" level=info msg="Testing certificate renew..." providerName=le.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2022-11-26T14:13:55+01:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
rwjack commented 2 years ago

Okay, changing traefik logs to debug as well just printed out a bunch of stuff, but nothing seems relevant. Are you guys having trouble replicating this issue?

Could it be my other middleware interfering somehow:

    security-headers:
      headers:
        referrerPolicy: same-origin
        forceSTSHeader: true
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        contentTypeNosniff: true
        browserXssFilter: true
        customRequestHeaders:
          X-Forwarded-Proto: https
        frameDeny: true
rwjack commented 2 years ago

I just want to make sure we all understand each other:

  clientTrustedIPs:
    - 10.0.0.0/24

The above config, as I understood, should work in the following manner:

Expected Case 1:

  1. 10.0.0.5 HTTP GET https://some.url.behind.traefik
  2. (Live) Plugin -x-> Crowdsec Container: No need to check with Crowdsec, because IP is in clientTrustedIPs
  3. (Stream) Plugin -x> Crowdsec Container: No need to check with Crowdsec, because IP is in clientTrustedIPs
  4. Traefik Returns 200

Expected Case 2:

  1. 195.100.150.202 (<- this is an external IP) HTTP GET https://some.url.behind.traefik
  2. (Live) Plugin -> Crowdsec Container: NEED to check with Crowdsec, because IP is NOT in clientTrustedIPs
  3. (Stream) Plugin -x> Crowdsec Container: No need to check with Crowdsec, because the plugin has a cache updated every 60s
  4. Check Crowdsec return (Live) | or local cache (Stream): Is that IP malicious?
  5. Traefik Returns 200 if IP is friendly, 403 if it's not

^ What actually happens in Case 2:

  1. ?
  2. ?
  3. Traefik Returns 403 anyways

Maybe this is a problem with the cache? Is it written to memory or a file? Maybe Traefik can't write the cache file?

mathieuHa commented 2 years ago

that I'm running Traefik on Debian, not as a container.

That's good to know!

but maybe better to use the A class as an example (which I actually use), so 10.0.0.0/24.

Did you meant a class C (containing 256 IP) with the identifier 10.0.0.X ?

Network: 10.0.0.0
CIDR: 24
Mask: 255.255.255.0
Range: 10.0.0.0 - 10.0.0.255
Total addresses in range: 256

Technically, yes, but not bound in the container since I'm running traefik on Debian with systemd. This entire git repo of this plugin is located in: /opt/traefik/plugins-local/src/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/

Alright that's clear, could you try to pin the local repository to the version 1.1.4 ?
https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/releases/tag/v1.1.4 Until we release we might discover some bugs on the main branch

I have added the following in the dynamic conf:

http:
middlewares:
crowdsec:
plugin:
bouncer:
...
logLevel: DEBUG
...

Can you please check the indentation ? Settings should be 2 char from the name of the plugin like this:

http:
  middlewares:
    crowdsec:
      plugin:
        bouncer:
          ...
          logLevel: DEBUG
          ...

For an exemple, I currently uses on my main proxy:

http:
  middlewares:
    crowdsec-demo: 
      plugin:
        bouncer:
          enabled: true
          crowdseclapikey: KEEEEEEEEEEEEEEY-demo
          updateintervalseconds: 60
          crowdsecmode: stream
          logLevel: "DEBUG"
          clientTrustedips: MY_PERSONAL_IP/32

Do I also need to change this in the static conf:

log: filePath: /var/log/traefik/traefik.log level: INFO # -> DEBUG?

I'm not sure, for the moment if we have to look at the debug logs of Traefik, the plugin should be logging independantly of Traefik log level

Because the logs don't show much:

That "may" be a good thing, at least no error visible, but clearly debug logs are not activated on the plugin, we have to find out why

Okay, changing traefik logs to debug as well just printed out a bunch of stuff, but nothing seems relevant. Are you guys having trouble replicating this issue?

I'll setup a demo environnement with a VM without container. Can you tell me which version of debian are you using ? 10, 11 ? Could you also share this log file (a few requests)

Could it be my other middleware interfering somehow:

I do not believe it's connected, your other middleware is only changing headers

The above config, as I understood, should work in the following manner:

Yes, you're right, I'll check on my side if everything is working as intended

Maybe this is a problem with the cache? Is it written to memory or a file? Maybe Traefik can't write the cache file? I have not thought about this, i'll respond after digging

Summary: I tried to answer all question, and ask you new informations Right now, we need to activate debug logs on the plugin, this should really guide us. We added a lot of logging through each step the plugin takes.

Thanks for taking the time to debug this with us!

rwjack commented 2 years ago

Did you meant a class C (containing 256 IP) with the identifier 10.0.0.X ?

Ugh, I'm trying to reveal as less information as I can, but I'll just be direct now to ease the debugging process. I use 10.0.0.0/8 - The literal class A, as the parameter for clientTrustedIPs.

Can you please check the indentation ? Settings should be 2 char from the name of the plugin like this:

My bad, indentation was good in the traefik config file, I edited it wrongly here.

I'm not sure, for the moment if we have to look at the debug logs of Traefik, the plugin should be logging independantly of Traefik log level

Okay, so where does it send logs to?

is the public IP really is the one the plugin receives

This could be related to the issue.

My current config: client request -> DMZ traefik VM (with this plugin) -> docker hypervisor (with crowdsec container)

rwjack commented 2 years ago

HEY @mathieuHa

is there a bug in the main branch not seen in release 1.1.4

This might be it! I just did git checkout v1.1.4, set ClientTrustedIPs to the A class, and I'm not getting 403s externally anymore. Checked from my phone.

I guess this narrows the search to v1.1.4 <-> Main

https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/compare/v1.1.4...main

mathieuHa commented 2 years ago

HEY @mathieuHa

is there a bug in the main branch not seen in release 1.1.4

This might be it! I just did git checkout v1.1.4, set ClientTrustedIPs to the A class, and I'm not getting 403s externally anymore. Checked from my phone.

I guess this narrows the search to v1.1.4 <-> Main

v1.1.4...main

Thanks for the feedback @rwjack Please let me know if you get any error with version 1.1.4 I'll work on fixing the problem before we can release 1.1.5

mathieuHa commented 2 years ago
  • The literal class A, as the parameter for clientTrustedIPs.

You can add multiples classes if you want in the parameter clusterTrustedIPs (it expects an array of IP/CIDR), but I know it can be hard to keep track of all ranges of IPs

rwjack commented 2 years ago

Please let me know if you get any error with version 1.1.4

Will do, thank you for helping out to get through this.

Guess this is a lesson for me to sometimes stick to releases, instead of going for bleeding edge all the time :)

mathieuHa commented 2 years ago

I think I found the guilty line of code

image

We did a major cleaning and rewrite last week and this condition has flipped.

rwjack commented 2 years ago

Awesome! Glad we got it sorted