How does Live-mode work?

Allram commented 1 year ago

So, this is not a bug-report :)

But i'm strugling a bit on how Live-mode actually work compared to Stream-mode.

When i run my instance in stream-mode (with Redis, thanks for adding the password!), Crowdsec will block IP's when they do something they shouldn't according to the collections i run. This seems to work great.

But when i run my Traefik instance in Live-mode, it never blocks any IP's. But i see that all of the IP's trying to connect to my instance gets a cache:hit, but maybe they are never forwarded to Crowdsec when they are hit in the cache, so that the buckets in Crowdsec will never fill up?

So, when i run Live-mode, will this plugin still send the IP's trying to connect to Crowdsec, or is that only every 60sec? In the latter case, that means it will "never" be triggered by Crowdsec for a ban?

mathieuHa commented 1 year ago

Hi @Allram

I'll explain tomorrow, and I will clarify live mode in the README. "Spoiler", it's a the same as none mode, but with a one minute cache. Il will also check that it is still working as intended.

Best, Mathieu

mathieuHa commented 1 year ago

Hey,

To reproduce and understand live mode, you may:

git clone the main branch of the project then

I set up the docker-compose.local.yml from the repository with the following edited to enable debug + live mode:

  whoami-foo:
    image: traefik/whoami
    container_name: "simple-service-foo"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.router-foo.rule=Path(`/foo`)"
      - "traefik.http.routers.router-foo.entrypoints=web"
      - "traefik.http.routers.router-foo.middlewares=crowdsec-foo@docker" 
      - "traefik.http.services.service-foo.loadbalancer.server.port=80"
      - "traefik.http.middlewares.crowdsec-foo.plugin.bouncer.enabled=true"
      - "traefik.http.middlewares.crowdsec-foo.plugin.bouncer.LogLevel=DEBUG"

      - "traefik.http.middlewares.crowdsec-foo.plugin.bouncer.CrowdsecMode=live"

      - "traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdseclapikey=40796d93c2958f9e58345514e67740e5"

Then run

docker compose -f docker-compose.local.yml up -d

I finally run docker compose logs -f to see all containers logs

docker compose -f docker-compose.local.yml logs -f

Requests are made with: GET http://localhost/bar

First request

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:15 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:15 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:15 ServeHTTP:getDecision ip:172.19.0.1 isBanned:false cache:miss

Not in cache so we query crowdsec (see crowdsec logs)

crowdsec            | time="10-03-2023 18:00:15" level=info msg="172.19.0.5 - [Fri, 10 Mar 2023 18:00:15 UTC] \"GET /v1/decisions?ip=172.19.0.1&banned=true HTTP/1.1 200 4.792311ms \"Go-http-client/1.1\" \""

We add the IP in the cache

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:15 cache:SetDecision ip:172.19.0.1 isBanned:false duration:60s
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:15 ServeHTTP:handleNoStreamCache ip:172.19.0.1 isBanned:false

Second request, IP is already in the cache

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:24 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:24 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:00:24 ServeHTTP ip:172.19.0.1 cache:hit isBanned:false

Then after a few minutes, same request, ip was not in cache

9e3420c324caabd50b7074dd77\" \""
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:05:15 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:05:15 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:05:15 ServeHTTP:getDecision ip:172.19.0.1 isBanned:false cache:miss
crowdsec            | time="10-03-2023 18:05:15" level=info msg="172.19.0.5 - [Fri, 10 Mar 2023 18:05:15 UTC] \"GET /v1/decisions?ip=172.19.0.1&banned=true HTTP/1.1 200 4.504014ms \"Go-http-client/1.1\" \""
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:05:15 cache:SetDecision ip:172.19.0.1 isBanned:false duration:60s
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 18:05:15 ServeHTTP:handleNoStreamCache ip:172.19.0.1 isBanned:false

Don't hesitate if you have any more question, and if you wanna be alerted when new version arrives, consider starring at/the project ;)

Allram commented 1 year ago

Thank you for the detailed explanation.

But as of my understanding, if the IP is cached, then crowdsec will not get notified? So t.ex the buckets in Crowdsec will not fill up, as the IP will only be sent to Crowdsec once if it's not in the cache?

I have ran this plugin now in Stream mode for 1 day, and i get alot of blocked IP's, when i run the plugin in Live mode, i have 0 blocked IP's. It just seems weird to me 🙈

Already starring the repo btw and have notification to discord via newreleases.io 👌

mathieuHa commented 1 year ago

Well Crowdsec is not notified by the plugin (Traefik).

It's the plugin that queries the decisions Crowdsec takes.

Decisions in Crowdsec are based on logs, and in our case Traefik logs.

So it's important to setup Crowdsec to read/monitor Traefik logs.
Then the behavior in the logs will impact who gets bans based on scenarios defined in crowdsec.

It's also very important to make sure you can see the original IP in Traefik logs, otherwise, Crowdsec might ban you're reverse proxy addr, or the one forwarded.
You can look for one use case where you need to setup something special in the exemples/behind-proxy directory of the repository.

On the logs I provide, it is setup in localhost with docker desktop, and not showing my true IP, check your Traefik logs to see if you see the True IP when setup in "production".

We can continue "debugging" with the following:

Go in the Crowdsec container:

docker exec -it crowdsec bash

Then you can play with cscli and look for instances at the decisions (here local env with no decisions yet)

bash-5.1# cscli decisions list
No active decisions

Again looking at the logs, we query http://localhost/bar

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 ServeHTTP:getDecision ip:172.19.0.1 isBanned:false cache:miss
crowdsec            | time="10-03-2023 19:16:28" level=info msg="172.19.0.5 - [Fri, 10 Mar 2023 19:16:28 UTC] \"GET /v1/decisions?ip=172.19.0.1&banned=true HTTP/1.1 200 3.977845ms \"Go-http-client/1.1\" \""

Now, in the crowdsec container we manually ban the our IP

bash-5.1# cscli decisions add --ip 172.19.0.1 -d 60s
INFO[10-03-2023 07:16:19 PM] Decision successfully added

We check that the decision is banned

bash-5.1# cscli decisions list
+----+--------+---------------+------------------------------+--------+---------+----+--------+-------------+----------+
| ID | SOURCE |  SCOPE:VALUE  |            REASON            | ACTION | COUNTRY | AS | EVENTS | EXPIRATION  | ALERT ID |
+----+--------+---------------+------------------------------+--------+---------+----+--------+-------------+----------+
|  1 | cscli  | Ip:172.19.0.1 | manual 'ban' from 'crowdsec' | ban    |         |    |      1 | 56.8388654s |        1 |
+----+--------+---------------+------------------------------+--------+---------+----+--------+-------------+----------+

Again looking at the logs, we query http://localhost/bar

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 ServeHTTP:getDecision ip:172.19.0.1 isBanned:false cache:miss
crowdsec            | time="10-03-2023 19:16:28" level=info msg="172.19.0.5 - [Fri, 10 Mar 2023 19:16:28 UTC] \"GET /v1/decisions?ip=172.19.0.1&banned=true HTTP/1.1 200 3.977845ms \"Go-http-client/1.1\" \""
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 cache:SetDecision ip:172.19.0.1 isBanned:true duration:50s
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:28 ServeHTTP:handleNoStreamCache ip:172.19.0.1 isBanned:true handleNoStreamCache:banned

Crowdsec responded it is banned for 50 sec, we register that in the cache (for 50 sec or 60sec I'm not sure)

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:29 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:29 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:16:29 ServeHTTP ip:172.19.0.1 cache:hit isBanned:true

Each following requests will not go through Crowdsec but the cache blocking it.

After a min, cache is invalidated and the next request will ask Crowdsec.

traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:35:25 ServeHTTP ip:172.19.0.1 isTrusted:false
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:35:25 cache:GetDecision ip:172.19.0.1
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:35:25 ServeHTTP:getDecision ip:172.19.0.1 isBanned:false cache:miss
crowdsec            | time="10-03-2023 19:35:25" level=info msg="172.19.0.5 - [Fri, 10 Mar 2023 19:35:25 UTC] \"GET /v1/decisions?ip=172.19.0.1&banned=true HTTP/1.1 200 3.511345ms \"Go-http-client/1.1\" \""
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:35:25 cache:SetDecision ip:172.19.0.1 isBanned:false duration:60s
traefik             | DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/10 19:35:25 ServeHTTP:handleNoStreamCache ip:172.19.0.1 isBanned:false

Also check that Crowdsec is reading correctly Traefik logs, and you have the collection crowdsecurity/traefik enabled in crowdsec:

bash-5.1# cscli collections list
WARN[10-03-2023 07:38:23 PM] Crowdsec is not the latest version. Current version is 'v1.4.1' and the latest stable version is 'v1.4.6'. Please update it! 
WARN[10-03-2023 07:38:23 PM] As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.4.6 
COLLECTIONS
------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH                                         
------------------------------------------------------------------------------------------------------------
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.6      /etc/crowdsec/collections/base-http-scenarios.yaml 
 crowdsecurity/http-cve             ✔️  enabled  1.1      /etc/crowdsec/collections/http-cve.yaml            
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml               
 crowdsecurity/sshd                 ✔️  enabled  0.2      /etc/crowdsec/collections/sshd.yaml                
 crowdsecurity/traefik              ✔️  enabled  0.1      /etc/crowdsec/collections/traefik.yaml             
------------------------------------------------------------------------------------------------------------
bash-5.1#

With the following cmd, you can check if Crowdsec has parsed your logs and what happens with the buckets

bash-5.1# cscli metrics
INFO[10-03-2023 07:39:29 PM] Acquisition Metrics:                         
+----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE              | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/traefik/access.log | 276        | 276          | -              | -                      |
+----------------------------------+------------+--------------+----------------+------------------------+
INFO[10-03-2023 07:39:29 PM] Parser Metrics:                              
+----------------------------------+------+--------+----------+
|             PARSERS              | HITS | PARSED | UNPARSED |
+----------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs    | 828  | 652    | 176      |
| child-crowdsecurity/traefik-logs | 276  | 276    | -        |
| crowdsecurity/dateparse-enrich   | 276  | 276    | -        |
| crowdsecurity/geoip-enrich       | 276  | 276    | -        |
| crowdsecurity/http-logs          | 276  | 275    | 1        |
| crowdsecurity/non-syslog         | 276  | 276    | -        |
| crowdsecurity/traefik-logs       | 276  | 276    | -        |
| crowdsecurity/whitelists         | 276  | 276    | -        |
+----------------------------------+------+--------+----------+
...

Let me know if that's more clear for you.
Best, Mathieu

Allram commented 1 year ago

Thank you for the detailed information :)

It seems like i was thinking that nothing got blocked because i didn't get Discord notifications of block happening while on Live mode. But that makes sense with what you wrote over here that the IP's actually get's blocked on the Traefik-Docker, before they get sent to Crowdsec :)

Allram commented 1 year ago

I think I actually found a bug. If i change the config of the plugin in Traefik or if Crowdsec Docker is started after the Traefik Docker, then Crowdsec will not be asked for decisions. (Atleast the startup sequence of the plugin is not working)

So if i change settings for the Traefik plugin, i have to actually restart Crowdsec Docker for the plugin to function correctly

mathieuHa commented 1 year ago

Hi @Allram,

We'll take a look and try to reproduce during the following week: I understood the following sequence

Traefik started with none mode
Crowdsec started

Traefik ask Crowdsec at every query
Traefik config changed to live mode

Traefik doesn't ask Crowdsec for any query

Is that what you are experiencing ?
If not, could you provide a sequence with previous config / following config and start/stop order so I can reproduce easily.

Thanks

Allram commented 1 year ago

Hello,

It's like this:

Traefik started in Stream mode (with or without RedisCache)
Crowdsec started Traefik ask Crowdsec at and does block if IP does something it should not/is on blocklist.
Crowdsec started
Traefik started in Stream mode (with or without RedisCache) Traefik ask Crowdsec at and does block if IP does something it should not/is on blocklist.
Traefik started in Live mode (with or without RedisCache)
Crowdsec started Traefik does not ask Crowdsec for every query and does not block any IP.
Crowdsec started
Traefik started in Live mode (with or without RedisCache) Traefik asks Crowdsec and block IP. (as it should)

When Crowdsec and Traefik works as they should, and i change config from Stream to Live, it stops working until i restart Crowdsec. The log says it's checking the IP/Decisions, but the log in Crowdsec only shows heartbeats.

It seems like the problem might be with the heartbeat/CheckHealth?

mathieuHa commented 1 year ago

Hi,

After a quick invest I opened another issue for something unrelated but that is a "bug"/improvement to do in the plugin.

So I start Traefik Crowdsec, it is healthy in stream mode.

In that case the plugin query once for all initial blocklist and then every minute or so for updates. It fetch the stream update of decisions.

At start of plugin

time="12-03-2023 10:11:56" level=info msg="172.21.0.4 - [Sun, 12 Mar 2023 10:11:56 UTC] \"GET /v1/decisions/stream?startup=true HTTP/1.1 200 17.744973ms \"Go-http-client/1.1\" \""

Every minutes

Crowdsec logs

TTP/1.1 200 3.816585ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:11:56" level=info msg="172.21.0.4 - [Sun, 12 Mar 2023 10:11:56 UTC] \"GET /v1/decisions/stream?startup=true HTTP/1.1 200 17.744973ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:12:55" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:12:55 UTC] \"GET /v1/heartbeat HTTP/1.1 200 11.815705ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:12:56" level=info msg="172.21.0.4 - [Sun, 12 Mar 2023 10:12:56 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 4.839833ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:13:55" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:13:55 UTC] \"GET /v1/heartbeat HTTP/1.1 200 5.563461ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:13:56" level=info msg="172.21.0.4 - [Sun, 12 Mar 2023 10:13:56 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 10.004651ms \"Go-http-client/1.1\" \""

Traefik logs

DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:17:56 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:18:56 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:18:56 cache:SetDecision ip:updated isBanned:false duration:59s

We will never see queries for a specific IP, only updates.

If I then switch to live mode
docker compose -f docker-compose.local.yml up -d

All logs trame:

DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:22 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:22 New initialized mode:stream
time="2023-03-12T10:23:22Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-03-12T10:23:39Z" level=debug msg="Provider event received {Status:die ID:b74972eb975113a26a09455925a5a0171319612746b175adda7b07c9b143ecb3 From:traefik/whoami Type:container Action:die Actor:{ID:b74972eb975113a26a09455925a5a0171319612746b175adda7b07c9b143ecb3 Attributes:map[com.docker.compose.config-hash:b02b65441240c682f3988bf5c97964f471cac8cd02c2121bdc397123aadada0d com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:420d3c2af3570ad1ced363bce5b284a4c662cf7ee178e261ae42b7d750d488a4 com.docker.compose.oneoff:False com.docker.compose.project:crowdsec-bouncer-traefik-plugin com.docker.compose.project.config_files:/home/mathieu/projets/crowdsec-bouncer-traefik-plugin/docker-compose.local.yml com.docker.compose.project.working_dir:/home/mathieu/projets/crowdsec-bouncer-traefik-plugin com.docker.compose.service:whoami-foo com.docker.compose.version:2.15.1 desktop.docker.io/wsl-distro:Ubuntu exitCode:2 image:traefik/whoami name:simple-service-foo org.opencontainers.image.created:2022-10-15T13:46:19Z org.opencontainers.image.description:Tiny Go webserver that prints OS information and HTTP request to output org.opencontainers.image.revision:e59736736851ca955cf8734fe4ea6d0c71ddbd62 org.opencontainers.image.source:https://github.com/traefik/whoami org.opencontainers.image.title:whoami org.opencontainers.image.version:1.8.7 traefik.enable:true traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdsecLapiHost:crowdsec:8080 traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdsecMode:stream traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdseclapikey:40796d93c2958f9e58345514e67740e5 traefik.http.middlewares.crowdsec-foo.plugin.bouncer.enabled:true traefik.http.middlewares.crowdsec-foo.plugin.bouncer.loglevel:DEBUG traefik.http.routers.router-foo.entrypoints:web traefik.http.routers.router-foo.middlewares:crowdsec-foo@docker traefik.http.routers.router-foo.rule:Path(`/foo`) traefik.http.services.service-foo.loadbalancer.server.port:80]} Scope:local Time:1678616619 TimeNano:1678616619687895043}" providerName=docker
time="2023-03-12T10:23:39Z" level=debug msg="Filtering disabled container" providerName=docker container=traefik-crowdsec-bouncer-traefik-plugin-d44b762fca4793c5ef6d5811f9cbd0207fad2ddc4f613ca96add1ae0bb7b2afe
time="2023-03-12T10:23:39Z" level=debug msg="Filtering disabled container" providerName=docker container=crowdsec-crowdsec-bouncer-traefik-plugin-80c9bbde813ec41a6a319db8160e56403c79602fb37d75ff1d595309d5fc5691
time="2023-03-12T10:23:39Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-03-12T10:23:40Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-03-12T10:23:40Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-03-12T10:23:40Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2023-03-12T10:23:40Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2023-03-12T10:23:40Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2023-03-12T10:23:40Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-03-12T10:23:40Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2023-03-12T10:23:40Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2023-03-12T10:23:40Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2023-03-12T10:23:40Z" level=debug msg="Provider event received {Status:start ID:f14b5cdcdaba6d6a6544fb58eaaa89055f302fd03f896b196f9d7dcf984ace16 From:traefik/whoami Type:container Action:start Actor:{ID:f14b5cdcdaba6d6a6544fb58eaaa89055f302fd03f896b196f9d7dcf984ace16 Attributes:map[com.docker.compose.config-hash:2416f594e8dce35980057c9ec74d0e1d7be81e031decc33904b92acb8467d131 com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:420d3c2af3570ad1ced363bce5b284a4c662cf7ee178e261ae42b7d750d488a4 com.docker.compose.oneoff:False com.docker.compose.project:crowdsec-bouncer-traefik-plugin com.docker.compose.project.config_files:/home/mathieu/projets/crowdsec-bouncer-traefik-plugin/docker-compose.local.yml com.docker.compose.project.working_dir:/home/mathieu/projets/crowdsec-bouncer-traefik-plugin com.docker.compose.service:whoami-foo com.docker.compose.version:2.15.1 desktop.docker.io/wsl-distro:Ubuntu image:traefik/whoami name:simple-service-foo org.opencontainers.image.created:2022-10-15T13:46:19Z org.opencontainers.image.description:Tiny Go webserver that prints OS information and HTTP request to output org.opencontainers.image.revision:e59736736851ca955cf8734fe4ea6d0c71ddbd62 org.opencontainers.image.source:https://github.com/traefik/whoami org.opencontainers.image.title:whoami org.opencontainers.image.version:1.8.7 traefik.enable:true traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdsecLapiHost:crowdsec:8080 traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdsecMode:live traefik.http.middlewares.crowdsec-foo.plugin.bouncer.crowdseclapikey:40796d93c2958f9e58345514e67740e5 traefik.http.middlewares.crowdsec-foo.plugin.bouncer.enabled:true traefik.http.middlewares.crowdsec-foo.plugin.bouncer.loglevel:DEBUG traefik.http.routers.router-foo.entrypoints:web traefik.http.routers.router-foo.middlewares:crowdsec-foo@docker traefik.http.routers.router-foo.rule:Path(`/foo`) traefik.http.services.service-foo.loadbalancer.server.port:80]} Scope:local Time:1678616620 TimeNano:1678616620551798983}" providerName=docker
time="2023-03-12T10:23:40Z" level=debug msg="Filtering disabled container" providerName=docker container=traefik-crowdsec-bouncer-traefik-plugin-d44b762fca4793c5ef6d5811f9cbd0207fad2ddc4f613ca96add1ae0bb7b2afe
time="2023-03-12T10:23:40Z" level=debug msg="Filtering disabled container" providerName=docker container=crowdsec-crowdsec-bouncer-traefik-plugin-80c9bbde813ec41a6a319db8160e56403c79602fb37d75ff1d595309d5fc5691
time="2023-03-12T10:23:41Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"router-foo\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"crowdsec-foo@docker\"],\"service\":\"service-foo\",\"rule\":\"Path(`/foo`)\"}},\"services\":{\"service-foo\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.22.0.3:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"crowdsec-foo\":{\"plugin\":{\"bouncer\":{\"crowdsecLapiHost\":\"crowdsec:8080\",\"crowdsecMode\":\"live\",\"crowdseclapikey\":\"40796d93c2958f9e58345514e67740e5\",\"enabled\":\"true\",\"loglevel\":\"DEBUG\"}}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-03-12T10:23:42Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-03-12T10:23:42Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal entryPointName=traefik middlewareName=tracing middlewareType=TracingForwarder
time="2023-03-12T10:23:42Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
time="2023-03-12T10:23:42Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2023-03-12T10:23:42Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2023-03-12T10:23:42Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-03-12T10:23:42Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-03-12T10:23:42Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-03-12T10:23:42Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-03-12T10:23:42Z" level=debug msg="Creating middleware" routerName=router-foo@docker serviceName=service-foo middlewareName=pipelining middlewareType=Pipelining entryPointName=web
time="2023-03-12T10:23:42Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=router-foo@docker serviceName=service-foo
time="2023-03-12T10:23:42Z" level=debug msg="Creating server 0 http://172.22.0.3:80" serverName=0 routerName=router-foo@docker serviceName=service-foo entryPointName=web
time="2023-03-12T10:23:42Z" level=debug msg="child http://172.22.0.3:80 now UP"
time="2023-03-12T10:23:42Z" level=debug msg="Propagating new UP status"
time="2023-03-12T10:23:42Z" level=debug msg="Added outgoing tracing middleware service-foo" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=router-foo@docker
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:42 No IP provided for ForwardedHeadersTrustedIPs
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:42 No IP provided for ClientTrustedIPs
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:42 getTLSConfigCrowdsec:CrowdsecLapiScheme https:no
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:42 cache:New initialized isRedis:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:23:42 New initialized mode:live
time="2023-03-12T10:23:42Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:24:22 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:24:22 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:24:22 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:25:22 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:25:22 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:25:22 handleStreamCache:updated

DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:22 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:22 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:22 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:37 ServeHTTP ip:172.22.0.1 isTrusted:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:37 cache:GetDecision ip:172.22.0.1
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:37 ServeHTTP ip:172.22.0.1 cache:hit isBanned:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:38 ServeHTTP ip:172.22.0.1 isTrusted:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/03/12 10:27:38 cache:GetDecision ip:172.22.0.1

time="12-03-2023 10:23:22" level=warning msg="new IP address detected for bouncer 'TRAEFIK_1': 172.22.0.4 (old: 172.21.0.4)"
time="12-03-2023 10:23:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:23:22 UTC] \"GET /v1/decisions/stream?startup=true HTTP/1.1 200 10.149134ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:23:57" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:23:57 UTC] \"GET /v1/heartbeat HTTP/1.1 200 5.734345ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:24:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:24:22 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 5.011383ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:24:57" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:24:57 UTC] \"GET /v1/heartbeat HTTP/1.1 200 6.219707ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:25:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:25:22 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 5.201463ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:25:57" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:25:57 UTC] \"GET /v1/heartbeat HTTP/1.1 200 4.387602ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:26:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:26:22 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 4.518322ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:26:57" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:26:57 UTC] \"GET /v1/heartbeat HTTP/1.1 200 4.342889ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:27:19" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:27:19 UTC] \"GET /v1/decisions?ip=172.22.0.1&banned=true HTTP/1.1 200 1.000792ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:27:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:27:22 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 10.754631ms \"Go-http-client/1.1\" \""
time="12-03-2023 10:27:57" level=info msg="127.0.0.1 - [Sun, 12 Mar 2023 10:27:57 UTC] \"GET /v1/heartbeat HTTP/1.1 200 4.604232ms \"crowdsec/v1.4.6-linux-5f71037b40c498045e1b59923504469e2b8d0140\" \""
time="12-03-2023 10:28:22" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:28:22 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 200 9.717702ms \"Go-http-client/1.1\" \""

Here I can see that the plugin has asked crowdsec for one IP:

time="12-03-2023 10:27:19" level=info msg="172.22.0.4 - [Sun, 12 Mar 2023 10:27:19 UTC] \"GET /v1/decisions?ip=172.22.0.1&banned=true HTTP/1.1 200 1.000792ms \"Go-http-client/1.1\" \""

But even in live mode, we have calls from stream mode. That does not seem to impact live mode, but that is still a "bug" we should fix. I'll open another issue about that.

Allram commented 1 year ago

Thanks for checking this :) It's working fine for me now after i restart Traefik after a config change, but it would be fine if it worked when config was changed as well.

So please close this or keep it open for further investiation :)

mathieuHa commented 1 year ago

We'll keep this opened until we fix #91 and #92

And we'll see if that help resolve what you experience after that :)

mathieuHa commented 7 months ago

Closing following fix for #91 and close of #92 see https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/issues/92#issuecomment-2033586462

maxlerebourg / crowdsec-bouncer-traefik-plugin

How does Live-mode work? #90