cannot get bouncer to enforce bans

[cellulosa]

hi there and thank you for the great work with this!

I am running crowdsec and caddy via docker, and a series of locally-hosted domains reached both locally (/etc/hosts) and via a cloudflared tunnel. All latest versions.

I can access my containers fine on both LAN and externally, and if I enable and inspect Caddy's access logs I can see the correct X-Forwarded-For address in both cases. However, if I try and run docker exec crowdsec cscli decisions add --ip on that ip (and also /24 ranges) I can still access all domains fine.

I am guessing it has to do with the bouncer? Any idea what it could be?

As for my config, here's what have:

docker-compose:

  caddy:
    container_name: caddy
    image: caddy:latest
    build:
      context: ./caddy
      dockerfile: Dockerfile
    volumes:
      - ./caddy:/data
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
      - CROWDSEC_LOCAL_API_KEY=${CROWDSEC_LOCAL_API_KEY}
    ports:
      - 80:80
      - 443:443

  cloudflared:
    container_name: cloudflared
    image: cloudflare/cloudflared:latest
    command: tunnel --no-autoupdate run
    volumes:
      - ./cloudflared:/home/nonroot/.cloudflared:ro

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec
    volumes:
      - ${DOCKER_CONFIG_DIR}/crowdsec/data:/var/lib/crowdsec/data
      - ${DOCKER_CONFIG_DIR}/crowdsec/config:/etc/crowdsec
    environment:
      - COLLECTIONS=crowdsecurity/caddy

  my-container:
    container_name: my-container
    image: some/image
    ports:
      - 1234:1234

Caddyfile:

{
    crowdsec {
        api_url http://crowdsec:8080
        api_key {$CROWDSEC_LOCAL_API_KEY}
    }
    email my@email.com
        acme_dns cloudflare {$CLOUDFLARE_API_TOKEN}
}

my.domain.com {
    route {
        crowdsec
        reverse_proxy my-container:1234
    }
}

Caddy Dockerfile:

FROM caddy:builder AS builder

RUN xcaddy build \
    # Use Cloudflare for ACME DNS challenge
    --with github.com/caddy-dns/cloudflare \
    # for crowdsec bouncer
    --with github.com/hslatman/caddy-crowdsec-bouncer/http

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Cloudflared config.yml:

tunnel: my-tunnel-id
credentials-file: /home/nonroot/.cloudflared/my-tunnel-id.json

ingress:
  - hostname: my.domain.com
    service: https://caddy:443
    originRequest:
      originServerName: "my.domain.com"
  - service: http_status:404

thanks!

[hslatman]

Hey @cellulosa, do you see a different client IP vs. the X-Forwarded-For header value? I think you'll have to set the global trusted_proxies setting to allow the Docker IP range to act as a trusted proxy. Also see https://caddyserver.com/docs/json/apps/http/servers/trusted_proxies/ and https://github.com/hslatman/caddy-crowdsec-bouncer/issues/15.

[cellulosa]

heya @hslatman thanks for getting back.

yes I do see a different client_ip and that's always the local docker network ip.

I just played a bit with the settings and with this the Caddyfile

{
    ...
    servers {
        trusted_proxies static private_ranges
        client_ip_headers Cf-Connecting-Ip X-Forwarded-For
    }
}

I can now see the right client_ip being set, but the ban still doesn't work (though I can see it's being set if I run docker exec crowdsec cscli decisions list).

[cellulosa]

oh actually I think that did the trick - it's just that the ban isn't as quick or perhaps there was some caching happening, so on hitting refresh the page was still loading, but after a while it stopped! Then I removed the ban and after a while it started working again. I can also see the ban count via crowdsec cscli metrics.

Thank you!

[hslatman]

You can change the interval the bouncer uses to check for new decisions using the ticker_interval setting. By default the bouncer uses the streaming mode, which actually works using periodic polling. You could change it to, say, 10s; or something like that. If you want the ban to happen immediately, the live mode can be used, which will call the CrowdSec agent for every HTTP request. The benefit of the streaming mode is that IPs are kept in a cache, so that lookups are a bit faster than when using the live mode. So it's a tradeoff between speed and the most correct result.

[cellulosa]

ahh I see, thank you for the detailed explanation. I think I can live with the delay if that has the speed benefit.

On a side note, I can see the logging and banning happens automatically via the API, so I understand there is no need to produce access log on caddy and have these inspected with the crowdsec's crowdsecurity/caddy collection?

[hslatman]

There's benefit to feeding Caddy's logs into the CrowdSec agent, but the bouncer doesn't provide functionality to do so. It needs to be configured a part of CrowdSec's acquisition.

By feeding (all) Caddy's logs into CrowdSec, a misbehaving client (that is not yet banned) might get marked as a bad client by CrowdSec if it hits the limits of a scenario.

[cellulosa]

yep that's what I got from reading about crowdsec - namely the fact that there are the two main components: one to scan log and the other to do the banning.

But because I couldn't see any mention of logging in this repo I thought maybe that happens behing the scenes somehow?

So would you suggest adding to the Caddyfile something like:

my.domain.com {
crowdsec
reverse_proxy ...
log {
        output file /data/logs/caddy.log
    }
}

And then adding those logs to crowdsec to get them analysed with crowdsecurity/caddy?

[hslatman]

Yes, it needs something like that. I have a separate repo intended for example configuration(s) that I think is better suited for that example. I decided to move some parts to there to keep this repo (c)leaner, but it's in an early stage. Feel free to open an issue there and/or open a PR with an example 🙂

[cellulosa]

I shall do that! Many thanks again for your support with this

[cellulosa]

Just wanted to flag that it isn't recommended to use X-Forwarded-For for it is vulnerable to spoofing

[samumatic]

@cellulosa could you possibly share your fixed Caddyfile again?

I got it like this:

 {
    email caddy@email
    acme_dns desec {
        token xyz
    }
    crowdsec {
        api_key {$CROWDSEC_API_KEY}
        api_url http://crowdsec:8080
    }
    order crowdsec first
    servers {
        trusted_proxies static private_ranges
        client_ip_headers Cf-Connecting-Ip X-Forwarded-For
    }
}

*.domain.com {
    log {
        output file /var/log/caddy/access.log
    }

    encode gzip

    @url host url.domain.com
    handle @url {
        reverse_proxy container:9999
    }
[...]

But the manual ban decisions are not working for me. I checked for the client_ip, but it is the same as the remote_ip. After 15 min. i can still reach the website.

[cellulosa]

this may be helpful, although from a quick look your Caddyfile looks fine to me... 🤔

Are you trying accessing your website from outside your network?

[samumatic]

I will try to replicate your repo with some adjustments on another server, and will see if that works, thanks. I will report back.

Yes i am trying to access the domain from my mobile phone network, found the IP on a whatsmyip website and verified it in my caddy logs. Added a ban decision in the crowdsec container and waited a couple of minutes, but no effect for me.

[hslatman]

@samumatic try adding the crowdsec directive inside the server block too. I think this will work:

*.domain.com {
    log {
        output file /var/log/caddy/access.log
    }

        crowdsec

    encode gzip

    @url host url.domain.com
    handle @url {
        reverse_proxy container:9999
    }
[...]

That is what enables crowdsec for that specific server as an HTTP handler. The top level crowdsec directive is for configuring its credentials, and that configuration is shared amongst all crowdsec directives defined in server blocks.

[samumatic]

It works, thank you! I see the mistake I made, at the beginning I got the same error as in https://github.com/hslatman/caddy-crowdsec-bouncer/issues/45#issuecomment-2204272668. Fixed it by adding the HTTP handler in xcaddy, but somehow removed the handler from the wildcard section.