fail2ban not releasing ban after expiration

pittbull commented 3 years ago

I seem to have an issue where fail2ban reactives a previous ban on an ip when the same ip tries to access the site after ban expires.

Environment:

traefik 2.4.8
fail2ban 0.6.0

middelwares.toml
  [http.middlewares.my-fail2ban]
    [http.middlewares.my-fail2ban.plugin]
      [http.middlewares.my-fail2ban.plugin.fail2ban]
        [http.middlewares.my-fail2ban.plugin.fail2ban.blacklist]
          ip = [""]
        [http.middlewares.my-fail2ban.plugin.fail2ban.rules]
          action = ""
          actionAbuseipdb = ""
          backend = ""
          banaction = ""
          banactionAllports = ""
          bantime = "1m"
          chain = ""
          destemail = ""
          enabled = true
          fail2banAgent = ""
          filter = ""
          findtime = "1m"
          ignorecommand = ""
          logencoding = "UTF-8"
          maxretry = 4
          mode = ""
          mta = ""
          ports = "0:8000"
          protocol = ""
          sender = ""
          urlregexp = ""
          usedns = ""
        [http.middlewares.my-fail2ban.plugin.fail2ban.whitelist]
          ip = ["::1", "127.0.0.1", "192.168.1.0/24", "172.16.1.0/16"]

static config:

      - --pilot.token=<redacted>
      - --experimental.plugins.fail2ban.modulename=github.com/tommoulard/fail2ban
      - --experimental.plugins.fail2ban.version=v0.6.0

service config: - "traefik.http.routers.authelia-rtr.middlewares=my-fail2ban@file,chain-no-auth@file"

Traefik log after restart:

Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Whitelisted: '::1/128'
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Whitelisted: '127.0.0.1/32'
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Whitelisted: '192.168.1.0/24'
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Whitelisted: '172.16.0.0/16'
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Bantime: 1m0s
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Findtime: 1m0s
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: Ports range from 0 to 8000
Fail2Ban_config: 2021/04/05 08:57:45 restricted.go:51: FailToBan Rules : '{Xbantime:1m0s Xfindtime:1m0s Xurlregexp:[] Xmaxretry:4 Xenabled:true Xports:[0 8000]}'
Fail2Ban: 2021/04/05 08:57:45 restricted.go:52: Plugin: FailToBan is up and running

Triggering ban:

Fail2Ban: 2021/04/05 09:03:17 restricted.go:52: <external client ip> is in blacklist mode
Fail2Ban: 2021/04/05 09:03:21 restricted.go:52: <external client ip> is in blacklist mode
Fail2Ban: 2021/04/05 09:03:25 restricted.go:52: <external client ip> is in blacklist mode

Then waiting until bantime expires:

1 Fail2Ban: 2021/04/05 09:05:20 restricted.go:52: <external client ip> is now back in whitelist mode
2 Fail2Ban: 2021/04/05 09:05:33 restricted.go:52: <external client ip> is in blacklist mode
3 Fail2Ban: 2021/04/05 09:05:54 restricted.go:52: <external client ip> is in blacklist mode

When accessing Authelia after bantime expires (line 1) fail2ban correctly notifies that ip is back in whitelist mode. However, a new request (line 2) puts the client back in fail2ban mode. This does not seem right and I wonder where I have blundered in my setup?

a-schaefers commented 3 years ago

where I experience similar behavior in a different context, I'm getting banned no matter what my regexpurl is set to, and it unbans after a minute, but then with any page reload, bans again.

pittbull commented 3 years ago

Likely related to #23 (comment)

where I experience similar behavior in a different context, I'm getting banned no matter what my regexpurl is set to, and it unbans after a minute, but then with any page reload, bans again.

I agree that the ban should be lifted when bantime has passed so something is wrong, but isn't the nature of Fail2ban that a specific ip is blocked regardless of url it tries to access? My understanding is that a firewall (iptables and similar) is used and thereby blocking the ip specifically?

cledavid commented 3 years ago

Hello ! We are looking on the problem ! It can take some time so please don't mind the lapse of time. For what I can see in the source code, the ban should be released after the minute that you specified in the configuration.

When a ip is banned, it is banned on all the domain that are registered with fail2ban so trying to access localhost:8000/one or localhost:8000/two after being banned will only reset the banning time. We are using a table of ip with the time of their first call and a number of try. If the number of try is more than the treshold, the ip is banned. If time of the call - time of first time of calls >= findtime, the ip is debanned and should be able to access the url.

We are going to make 2 releases in the upcoming weeks :

more logs and loglevel (that will help us debug your case and upcoming ones)
regexp (allow, block, filter)

Thoses releases will help to at least debug your problem and if the problem lie in regexp it could be resolved like that ! We'll back soon with more solution so wait for us 😉 Clement

pittbull commented 3 years ago

Looking forward to your updates.

Met me know if you need any debug information.

cledavid commented 3 years ago

Hello,

We released two new versions of fail2ban : 0.6.1 & 0.6.2. 0.6.1 include an upgrade of regexp 0.6.2 add more logs You can upgrade your version of fail2ban and check if you're still experiencing your problems, you'll have to change a bit your rules but everything is in the README.

Clement

pittbull commented 3 years ago

How would this config look in my .toml file?

tomMoulard commented 3 years ago

https://github.com/tomMoulard/fail2ban/blob/main/README.md#fail2ban

pittbull commented 3 years ago

Ok.

Updated now but situation is still the same: fail2ban basically just banning when I access my site for no specific reason. 🤷🏼‍♂️

tomMoulard commented 3 years ago

Can you produce a reproducible case ?

for no specific reason

Is not enough for me do to do something about it.

pittbull commented 3 years ago

middlewares.toml

  [http.middlewares.my-fail2ban]
    [http.middlewares.my-fail2ban.plugin]
      [http.middlewares.my-fail2ban.plugin.fail2ban]
        logLevel= "INFO"
        [http.middlewares.my-fail2ban.plugin.fail2ban.rules]
          bantime= "10s"
          enabled= true
          findtime= "60m"
          maxretry= 3
          ports= "80:443"

       [http.middlewares.my-fail2ban.plugin.fail2ban.whitelist]
         ip = ["::1", "127.0.0.1", "192.168.1.0/24"]

Activation:

INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Whitelisted: '::1/128'
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Whitelisted: '127.0.0.1/32'
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Whitelisted: '192.168.1.0/24'
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Bantime: 10s
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Findtime: 1h0m0s
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: Ports range from 80 to 443
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:51: FailToBan Rules : '{Xbantime:10s Xfindtime:1h0m0s XurlregexpAllow:[] XurlregexpBan:[] Xmaxretry:3 Xenabled:true Xports:[80 443]}'
INFO: Fail2Ban: 2021/05/06 17:15:33 restricted.go:52: Plugin: FailToBan is up and running

Debug log:

DEBUG: Fail2Ban: 2021/05/06 17:19:21 restricted.go:51: New request: &{GET / HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8] Accept-Encoding:[gzip] Accept-Language:[en-us] Cdn-Loop:[cloudflare] Cf-Connecting-Ip:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] Cf-Ipcountry:[NO] Cf-Ray:[64b32fb7b9d60b6b-AMS] Cf-Request-Id:[09e3dc26d400000b6bf11bb000000001] Cf-Visitor:[{"scheme":"https"}] Connection:[Keep-Alive] Cookie:[authelia_session=vpjQsqHJtyMZtqUYWKpTQfeHuGrXrGpq; __cfduid=d4bcb3ac6988160f0689ba91f129051061620310071] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Mobile/15E148 Safari/604.1] X-Forwarded-For:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] X-Forwarded-Host:[<mysite>] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[a4aa097bf2e3] X-Real-Ip:[141.101.105.91]] 0xc001606600 <nil> 0 [] false <mysite> map[] map[] <nil> map[] 141.101.105.91:24982 / 0xc00140a000 <nil> <nil> 0xc000f06300}
DEBUG: Fail2Ban: 2021/05/06 17:19:21 restricted.go:52: 141.101.105.91 is no longer banned
DEBUG: Fail2Ban: 2021/05/06 17:19:21 restricted.go:51: New request: &{GET /web/index.html HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8] Accept-Encoding:[gzip] Accept-Language:[en-us] Cdn-Loop:[cloudflare] Cf-Connecting-Ip:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] Cf-Ipcountry:[NO] Cf-Ray:[64b32fb8bbad0b6b-AMS] Cf-Request-Id:[09e3dc277100000b6b140f6000000001] Cf-Visitor:[{"scheme":"https"}] Connection:[Keep-Alive] Cookie:[authelia_session=vpjQsqHJtyMZtqUYWKpTQfeHuGrXrGpq; __cfduid=d4bcb3ac6988160f0689ba91f129051061620310071] Referer:[https:// <mysite>/] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Mobile/15E148 Safari/604.1] X-Forwarded-For:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] X-Forwarded-Host:[ <mysite>] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[a4aa097bf2e3] X-Real-Ip:[141.101.105.91]] 0xc000ce44b0 <nil> 0 [] false  <mysite> map[] map[] <nil> map[] 141.101.105.91:36900 /web/index.html 0xc00064ea50 <nil> <nil> 0xc000ecb9b0}
DEBUG: Fail2Ban: 2021/05/06 17:19:21 restricted.go:51: welcome back 141.101.105.91 for the 2 time
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:51: New request: &{GET /web/translations/en.json HTTP/1.1 1 1 map[Accept:[application/json, text/javascript, */*; q=0.01] Accept-Encoding:[gzip] Accept-Language:[en-us] Cdn-Loop:[cloudflare] Cf-Connecting-Ip:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] Cf-Ipcountry:[NO] Cf-Ray:[64b32fba5e720b6b-AMS] Cf-Request-Id:[09e3dc287800000b6bc7971000000001] Cf-Visitor:[{"scheme":"https"}] Connection:[Keep-Alive] Cookie:[authelia_session=vpjQsqHJtyMZtqUYWKpTQfeHuGrXrGpq; __cfduid=d4bcb3ac6988160f0689ba91f129051061620310071] Referer:[https:// <mysite>/web/index.html] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Mobile/15E148 Safari/604.1] X-Forwarded-For:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] X-Forwarded-Host:[ <mysite>] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[a4aa097bf2e3] X-Real-Ip:[141.101.105.91] X-Requested-With:[XMLHttpRequest]] 0xc00194e618 <nil> 0 [] false  <mysite> map[] map[] <nil> map[] 141.101.105.91:45168 /web/translations/en.json 0xc0017e4000 <nil> <nil> 0xc000546e10}
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:52: 141.101.105.91 is now banned temporarily
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:51: New request: &{GET /media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.54.5&X-Plex-Client-Identifier=a3c11yljb5j8p0g74iucl5ww&X-Plex-Platform=Safari&X-Plex-Platform-Version=605.1&X-Plex-Sync-Version=2&X-Plex-Features=external-media%2Cindirect-media&X-Plex-Model=bundled&X-Plex-Device=iOS&X-Plex-Device-Name=Safari&X-Plex-Device-Screen-Resolution=980x1669%2C390x844&X-Plex-Token=52sp_sdeytDjRKtTf66q&X-Plex-Language=en HTTP/1.1 1 1 map[Accept:[application/json] Accept-Encoding:[gzip] Accept-Language:[en] Cdn-Loop:[cloudflare] Cf-Connecting-Ip:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] Cf-Ipcountry:[NO] Cf-Ray:[64b32fbc5a2b0b6b-AMS] Cf-Request-Id:[09e3dc29ba00000b6bd7892000000001] Cf-Visitor:[{"scheme":"https"}] Connection:[Keep-Alive] Cookie:[authelia_session=vpjQsqHJtyMZtqUYWKpTQfeHuGrXrGpq; __cfduid=d4bcb3ac6988160f0689ba91f129051061620310071] Referer:[https:// <mysite>/web/index.html] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Mobile/15E148 Safari/604.1] X-Forwarded-For:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] X-Forwarded-Host:[ <mysite>] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[a4aa097bf2e3] X-Real-Ip:[141.101.105.91] X-Requested-With:[XMLHttpRequest]] 0xc0016070c8 <nil> 0 [] false  <mysite> map[] map[] <nil> map[] 141.101.105.91:36900 /media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.54.5&X-Plex-Client-Identifier=a3c11yljb5j8p0g74iucl5ww&X-Plex-Platform=Safari&X-Plex-Platform-Version=605.1&X-Plex-Sync-Version=2&X-Plex-Features=external-media%2Cindirect-media&X-Plex-Model=bundled&X-Plex-Device=iOS&X-Plex-Device-Name=Safari&X-Plex-Device-Screen-Resolution=980x1669%2C390x844&X-Plex-Token=52sp_sdeytDjRKtTf66q&X-Plex-Language=en 0xc00064ea50 <nil> <nil> 0xc000f07650}
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:51: 141.101.105.91 is still banned since 2021-05-06T17:19:22+02:00, 4 request
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:51: New request: &{GET /media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.54.5&X-Plex-Client-Identifier=a3c11yljb5j8p0g74iucl5ww&X-Plex-Platform=Safari&X-Plex-Platform-Version=605.1&X-Plex-Sync-Version=2&X-Plex-Features=external-media%2Cindirect-media&X-Plex-Model=bundled&X-Plex-Device=iOS&X-Plex-Device-Name=Safari&X-Plex-Device-Screen-Resolution=980x1669%2C390x844&X-Plex-Language=en HTTP/1.1 1 1 map[Accept:[application/json] Accept-Encoding:[gzip] Accept-Language:[en] Cdn-Loop:[cloudflare] Cf-Connecting-Ip:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] Cf-Ipcountry:[NO] Cf-Ray:[64b32fbc6a4a0b6b-AMS] Cf-Request-Id:[09e3dc29c400000b6b16068000000001] Cf-Visitor:[{"scheme":"https"}] Connection:[Keep-Alive] Cookie:[authelia_session=vpjQsqHJtyMZtqUYWKpTQfeHuGrXrGpq; __cfduid=d4bcb3ac6988160f0689ba91f129051061620310071] Referer:[https:// <mysite>/web/index.html] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Mobile/15E148 Safari/604.1] X-Forwarded-For:[2a02:2121:30c:fc3f:ad2e:cbcb:b8f3:d799] X-Forwarded-Host:[ <mysite>] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[a4aa097bf2e3] X-Real-Ip:[141.101.105.91] X-Requested-With:[XMLHttpRequest]] 0xc0008b40d8 <nil> 0 [] false  <mysite> map[] map[] <nil> map[] 141.101.105.91:38766 /media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.54.5&X-Plex-Client-Identifier=a3c11yljb5j8p0g74iucl5ww&X-Plex-Platform=Safari&X-Plex-Platform-Version=605.1&X-Plex-Sync-Version=2&X-Plex-Features=external-media%2Cindirect-media&X-Plex-Model=bundled&X-Plex-Device=iOS&X-Plex-Device-Name=Safari&X-Plex-Device-Screen-Resolution=980x1669%2C390x844&X-Plex-Language=en 0xc00064ec60 <nil> <nil> 0xc000f062a0}
DEBUG: Fail2Ban: 2021/05/06 17:19:22 restricted.go:51: 141.101.105.91 is still banned since 2021-05-06T17:19:22+02:00, 5 request

tomMoulard commented 3 years ago

141.101.105.91 is not in ["::1", "127.0.0.1", "192.168.1.0/24"], so it works as expected

pittbull commented 3 years ago

Does that mean that the implementation of the fail2ban module is to ban all ip's not whitelisted? Because that is not what I expected based on my experience with it's namesake program.

tomMoulard commented 3 years ago

Here's the doc on how regexp works.

Feel free to open a pr that could enhance either the doc or the regexp (and the plugin) behaviour.

I will close this issue as it seems that you are asking us to build your configuration. But don't hesitate to ask your issue about it on the Traefik community forum.

tomMoulard / fail2ban

fail2ban not releasing ban after expiration #34