Need help setting up urlregexp rule

DaniSchenk commented 3 years ago

Hey everyone, I already reached out to the traefik community forum but couldn't find any help so far (post).

I'm trying to set up the fail2ban plugin and it works with my containous/whoami test container without a urlregexp rule.

I want to secure my Wordpress container by setting a urlregexp rule for the login page /wp-login.php. But it seems that no matter which rule I configure, fail2ban does not filter for the login page but takes all requests to the site for its calculations. This leads to access restrictions relatively fast on all Wordpress pages. My attempts so far (docker-compose of my Wordpress container):

    labels:
     [ ... ]
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.enabled=true"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.bantime=5m"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.findtime=1m"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.maxretry=5"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.ports=0:8000"

      # with and without \`
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp=`.*\/wp-login.php.*`"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp=.*\/wp-login.php.*"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp=.*\/wp\-login\.php.*" # throws compose parsing error

      - "traefik.http.routers.wordpress.middlewares=my-fail2ban@docker"
      [ ... ]

Is my regex not correct? Or am I mistaken what the urlregexp rule should do?

Thanks in advance

tomMoulard commented 3 years ago

Hi there !

First of all, you are right ! That is how we intended urlregexp to work ! If the url is a match for the regexp urlregexp, the ip would be banned.

In your case, I think that you can use:

- "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp='wp-login.php'"

Wildcard will be infered by Golang Regexp ! If you want to check your regexps, I sugest that you use regex101.

Be aware that there should be some logs if the url is matched with an ip like:

Fail2Ban: 2020/12/04 10:30:00 restricted.go:51: Url ('/wp-login.php') was matched by regexp: 'wp-login.php'

I hope it helps !

Tom

DaniSchenk commented 3 years ago

Thx for your fast reply and I'll give it a try tonight.

DaniSchenk commented 3 years ago

I think I got the regex right by removing the single quotes from your example:

- "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp=wp-login.php"

My log shows a match now:

Fail2Ban: 2020/12/04 18:20:30 restricted.go:51: Url ('/wp-login.php?test') was matched by regexp: 'wp-login.php'

But still, fai2ban considers all my requests. Not only the ones to domain.tld/wp-login.php. I get banned as soon as I load my main page (domain.tld) as well. It doesn't matter which url I'm requesting. It looks like fail2ban ignores my urlregexp rule.

tomMoulard commented 3 years ago

It is great if you get it working !

Yeah! The plugin work for the whole Traefik routing. Do you think that it will be interesting to filter with in a setting of each service ?

DaniSchenk commented 3 years ago

In my docker setup there are dozens of services running which are using traefik. E.g.:

domain1.tld node
domain2.tld nextcloud
domain3.tld wordpress

My goal is to protect only one single url of them: domain3.tld/wp-login.php But my current config (see above) is protecting all URLs of domain3.tld.

So yes, my goal is to limit fail2ban to a specific service (already working) and further more to a specific URL (not working). I thought the urlregexp rule could be used as a filter for specific URLs.

tomMoulard commented 3 years ago

Hi @DaniSchenk,

I've made this change: commit.

It adds support for multi regexp and now the url tested look like this: domain3.tld/wp-login.php (see rules-fail2ban.yaml).

What do you think fo it ? Does it solves your usecase ?

DaniSchenk commented 3 years ago

Hey @tomMoulard,

Adding support for adding multiple regex filters seems pretty usefull to me 👍.

I had problems to run the plugin including your commit. I have no experience with go or traefik plugins. On my Windows local maschine (Docker Desktop and WSL2) it tried:

created a directory structure /plugins/go/src/github.com/tommoulard/fail2ban and placed the source code including your commit in it
added some lines in my local traefik docker-compose.yml:
```
volumes:
[ ... ]
- "/plugins/go:/plugins/go"
command:
[ ... ]
- "--experimental.devPlugin.moduleName=github.com/tommoulard/fail2ban"
- "--experimental.devPlugin.goPath=/plugins/go"
```
I failed. My local traefik dashboard displays the error: plugin: unknown plugin type: fail2ban. So I didn't manage to test your changes.

Despite this, I'm not sure that your changes will address the issue I have. As I mentioned earlier (comment) I already got it working to get a match for my URL. It's just a wild guess, but I think that fail2ban rejects the request before it even considers a urlregexp check. It seems to me, that not all conditions must be met to reject a request. Even if the urlregexp filter is not matching, fail2ban considers the requesting IP to be banned.

cledavid commented 3 years ago

Hi,

We just released a new version of fail2ban, with more logs and the multiple regexp. v0.5.0 You can test the new functions here.

We we manage the flux sequencialy in this order :

Blacklist
Whitelist
Urlregxp
Fail2ban

Check if your ip is not banned by traefik in the logs and check if your ip is not in the blacklist before testing. If you have any other questions, don't hesitate !

DaniSchenk commented 3 years ago

I updated to the most recent version of your plugin and my configs seem to get parsed correctly:

Fail2Ban_config: 2021/01/07 23:49:05 restricted.go:51: Bantime: 5s
Fail2Ban_config: 2021/01/07 23:49:05 restricted.go:51: Findtime: 5s
Fail2Ban_config: 2021/01/07 23:49:05 restricted.go:51: Ports range from 80 to 443
Fail2Ban_config: 2021/01/07 23:49:05 restricted.go:51: FailToBan Rules : '{Xbantime:5s Xfindtime:5s Xurlregexp:[`.*/wp-login.php`] Xmaxretry:5 Xenabled:true Xports:[80 443]}'
Fail2Ban: 2021/01/07 23:49:05 restricted.go:52: Plugin: FailToBan is up and running

However, fail2ban is still banning all requests on all enpoints and is not respecting my urlregexp rules. After the 5th request to any endpoint I get this log entry:

Fail2Ban: 2021/01/07 23:49:41 restricted.go:52: xx.xxx.59.0:16041 is in blacklist mode

I tried a lot of different rules. E.g. '.*/wp-login.php', .*/wp-login, wp-login.php, \/wp-login\.php, wp-login, and much more ... When I define my rule using a starting * like */wp-login I get a log entry similar to:

Fail2Ban: 2021/01/07 23:57:25 restricted.go:51: Url ('domain.tld/') was matched by regexp: '*/wp-login'

But this rule blocks all requests on all endpoints imediately without even respecting any other like findtime

Can you give me a working urlregex rule, which only considers a specific endpoint? A rule that:

considers to block: domain.tld/path1
but never blocks:
- domain.tld/
- domain.tld/path2

cledavid commented 3 years ago

Hi,

We just found out that our parsing was not good... The ip that we were matching to the blacklist and whitelist were in fact "ip:port". We are right now trying to fix that. We are not full time on this and right now we don't have that much time for this project. I think that this will be fixed before the end of january. You can check the advencement on this branch : https://github.com/tomMoulard/fail2ban/tree/port-support Sorry for the delay and thanks for your patience.

Clement

cledavid commented 3 years ago

Hi,

We just released the 0.6.0 version of the fail2ban plugin. You can try this new version with your traefik Thanks for the wait and hope to ear from you about this release

Clement

DaniSchenk commented 3 years ago

Hey @cledavid, thanks for your update. My issue is still present.

I created a small repo to reproduce the issue https://github.com/DaniSchenk/fail2ban-bug. Check the README to get instructions. It's basically done by adding your token running docker-compose up.

cledavid commented 3 years ago

Hey @DaniSchenk, I just checked your repo and it is in fact working as expected. For now, urlregexp block the request as specified in the readme : urlregexp: a regexp list to block request with regexps on the url

We will soon add an option on the urlregexp to allow, block and filter. allow and block will be respectively white and blacklist for url filter will restrain the fail2ban to only these urlregexp

Moreover, if you have multiple services, you can do some urlregexp like : 'toto.com/path1' 'tata.com/path2' 't*.com/path3'

Clement

DaniSchenk commented 3 years ago

Oh, I see. I thought the urlregex config would behave like the failregex filter from fail2ban. I'm looking forward to your upcoming updates 👍

a-schaefers commented 3 years ago

@cledavid

I'm experiencing this same exact issue, it not worse. It doesn't match against the regexpurl, but it does keep track of an IP, and ends up blocking me for a minute if I reload a page X times, regardless of whether or not it's the wp-login.php which is the ONLY page that I want protected.

Configuration:

      - --pilot.token=************************
      - --experimental.plugins.fail2ban.modulename=github.com/tommoulard/fail2ban
      - --experimental.plugins.fail2ban.version=v0.6.0
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.enabled=true"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.bantime=1m"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.findtime=1m"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.maxretry=5"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.ports=0:8000"
      - "traefik.http.middlewares.my-fail2ban.plugin.fail2ban.rules.urlregexp=wp-login.php"
      - "traefik.http.routers.my-wordpress-site.middlewares=my-fail2ban@docker"

In the following logs, notice I am simply loading /index.php over and over again some ~10x and it results in a 1 minute IP ban.

-traefik | time="2021-04-06T02:28:12Z" level=info msg="Configuration loaded from flags." -traefik | Fail2Ban_config: 2021/04/06 02:28:17 restricted.go:51: Bantime: 1m0s -traefik | Fail2Ban_config: 2021/04/06 02:28:17 restricted.go:51: Findtime: 1m0s -traefik | Fail2Ban_config: 2021/04/06 02:28:17 restricted.go:51: Ports range from 0 to 8000 -traefik | Fail2Ban_config: 2021/04/06 02:28:17 restricted.go:51: FailToBan Rules : '{Xbantime:1m0s Xfindtime:1m0s Xurlregexp:[wp-login.php] Xmaxretry:5 Xenabled:true Xports:[0 8000]}' -traefik | Fail2Ban: 2021/04/06 02:28:17 restricted.go:52: Plugin: FailToBan is up and running -.com | 192.168.96.5 - - [06/Apr/2021:02:28:35 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.2750239372253417968750 HTTP/1.1" 200 31 "https://*****.com/wp-cron.php?doing_wp_cron=1617676115.2750239372253417968750" "WordPress/5.7; https://*****.com" -traefik | 192.168.96.1 - - [06/Apr/2021:02:28:35 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.2750239372253417968750 HTTP/1.1" 200 20 "-" "-" 2 "@docker" "http://192.168.96.6:8080" 4ms -.com | 192.168.96.5 - - [06/Apr/2021:02:28:35 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.6093220710754394531250 HTTP/1.1" 200 31 "https://*****.com/wp-cron.php?doing_wp_cron=1617676115.6093220710754394531250" "WordPress/5.7; https://*****.com" -traefik | 192.168.96.1 - - [06/Apr/2021:02:28:35 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.6093220710754394531250 HTTP/1.1" 200 20 "-" "-" 3 "@docker" "http://192.168.96.6:8080" 2ms -.com | 127.0.0.1 - 06/Apr/2021:02:28:35 +0000 "POST /wp-cron.php" 200 -.com | 192.168.96.5 - - [06/Apr/2021:02:28:36 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.8865261077880859375000 HTTP/1.1" 200 31 "https://*****.com/wp-cron.php?doing_wp_cron=1617676115.8865261077880859375000" "WordPress/5.7; https://*****.com" -traefik | 192.168.96.1 - - [06/Apr/2021:02:28:36 +0000] "POST /wp-cron.php?doing_wp_cron=1617676115.8865261077880859375000 HTTP/1.1" 200 20 "-" "-" 4 "@docker" "http://192.168.96.6:8080" 2ms -.com | 127.0.0.1 - 06/Apr/2021:02:28:32 +0000 "GET /index.php" 200 -.com | 192.168.96.5 - - [06/Apr/2021:02:28:36 +0000] "GET / HTTP/1.1" 200 14365 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" **-traefik | 67.204..* - - [06/Apr/2021:02:28:32 +0000] "GET / HTTP/2.0" 200 14352 "-" "-" 1 "***@docker" "http://192.168.96.6:8080" 3552ms -.com | 127.0.0.1 - 06/Apr/2021:02:28:35 +0000 "POST /wp-cron.php" 200 -.com | 127.0.0.1 - 06/Apr/2021:02:28:36 +0000 "POST /wp-cron.php" 200 -.com | 127.0.0.1 - 06/Apr/2021:02:28:36 +0000 "POST /index.php" 200 -.com | 192.168.96.5 - - [06/Apr/2021:02:28:37 +0000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 398 "https://*****.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" **-traefik | 67.204..* - - [06/Apr/2021:02:28:36 +0000] "POST /?wc-ajax=get_refreshed_fragments HTTP/2.0" 200 386 "-" "-" 5 "***@docker" "http://192.168.96.6:8080" 352ms -.com | 127.0.0.1 - 06/Apr/2021:02:28:45 +0000 "GET /index.php" 200 -.com | 192.168.96.5 - - [06/Apr/2021:02:28:45 +0000] "GET / HTTP/1.1" 200 14357 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" **-traefik | 67.204..* - - [06/Apr/2021:02:28:45 +0000] "GET / HTTP/2.0" 200 14344 "-" "-" 6 "***@docker" "http://192.168.96.6:8080" 389ms -traefik | crond: user:root entry:33 23 /usr/local/bin/update-blacklist.sh /etc/ipset-blacklist/ipset-blacklist.conf 2>&1 ****-traefik | 000000000000000000000000000000000100000000000000000000000000 -traefik | 000000000000000000000001 -traefik | 11111111111111111111111111111111 -traefik | 111111111111 -traefik | 1111111 -traefik | crond: wakeup dt=48 -traefik | crond: file root: -traefik | crond: line /usr/local/bin/update-blacklist.sh /etc/ipset-blacklist/ipset-blacklist.conf 2>&1 -.com | 127.0.0.1 - 06/Apr/2021:02:29:08 +0000 "GET /index.php" 200 -.com | 192.168.96.5 - - [06/Apr/2021:02:29:08 +0000] "GET / HTTP/1.1" 200 14357 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" **-traefik | 67.204..* - - [06/Apr/2021:02:29:08 +0000] "GET / HTTP/2.0" 200 14344 "-" "-" 7 "***@docker" "http://192.168.96.6:8080" 388ms **-traefik | Fail2Ban: 2021/04/06 02:29:11 restricted.go:52: 67.204..* is in blacklist mode ***-traefik | 67.204.. - - [06/Apr/2021:02:29:11 +0000] "GET / HTTP/2.0" 403 0 "-" "-" 8 "*****@docker" "-" 0ms

After another minute I'll get

**-traefik | Fail2Ban: 2021/04/06 02:40:25 restricted.go:52: 67.204..*** is now back in whitelist mode

But if I reload any page X times regardless of it being wp-login.php will result in a block.

cledavid commented 3 years ago

Hello, As found in the thread, the regexp is only blocking a part of the website without going through fail2ban. We are working on this and you can see some move in the new_urlregexp branch.

This new regexp will create 3 types :

allow (the call is not checked and every call pass)
block (the call is not checked and every call is stopped)
filter (the call is going through fail2ban)

It will come in the upcoming weeks, we are working on fail2ban on our free time and this last month was everything exept free so we didn't make a lot of improvement. 😓 We will do our best to release the new version as soon as possible 😉 Clement

tomMoulard commented 3 years ago

Closed by #35.

tomMoulard / fail2ban

Need help setting up urlregexp rule #23