The REGEX does not seem to work

[leoaloha]

Xbuntu 16.04, fail2ban v9, installed fail2ban-subnets as directed. when run against the fail2ban.log file there are zero hits, yet I have over 200 hits with my sshd filter. When i do a trace on /etc/cron.hourly/fail2ban-subnets, i get 39837 iterations reading the fail2ban.log.1 file at "for l in fh:" in the ### LOGIC section but only 1 iteration at "# Filter then sort the offenders by order or higher offense". there are 337 confirmed bans in this file

[XaF]

I would need to see an example of the log format to see if the regex is actually the problem. My fail2ban version is still 0.8.13 at the moment. Also, it's highly possible that fail2ban-subnets is just doing one iteration because other offenders were not classified in common subnets.

[leoaloha]

snip of log file 2016-07-17 07:53:33,029 fail2ban.filter [1814]: INFO Log rotation detected for /var/log/apache2/access.log 2016-07-17 08:24:48,324 fail2ban.filter [1814]: INFO [sshd] Found 188.0.236.149 2016-07-17 08:24:48,505 fail2ban.filter [1814]: INFO [sshd] Found 188.0.236.149 2016-07-17 08:24:50,229 fail2ban.filter [1814]: INFO [sshd] Found 188.0.236.149 2016-07-17 08:32:59,317 fail2ban.filter [1814]: INFO [sshd] Found 182.100.67.174 2016-07-17 08:33:01,119 fail2ban.filter [1814]: INFO [sshd] Found 182.100.67.174 2016-07-17 08:33:43,788 fail2ban.filter [1814]: INFO [sshd] Found 182.100.67.174 2016-07-17 08:33:45,629 fail2ban.filter [1814]: INFO [sshd] Found 182.100.67.174 2016-07-17 08:33:46,108 fail2ban.actions [1814]: NOTICE [sshd] Ban 182.100.67.174 2016-07-17 08:33:46,129 fail2ban.filter [1814]: INFO [fail2ban] Found 182.100.67.174 2016-07-17 08:43:46,255 fail2ban.actions [1814]: NOTICE [sshd] Unban 182.100.67.174 2016-07-17 09:53:48,250 fail2ban.filter [1814]: INFO [apache-badbots] Found 66.249.75.108 2016-07-17 09:53:48,502 fail2ban.filter [1814]: INFO [apache-fakegooglebot] Ignore 66.249.75.108 by command 2016-07-17 09:53:48,560 fail2ban.actions [1814]: NOTICE [apache-badbots] Ban 66.249.75.108 2016-07-17 09:53:48,668 fail2ban.filter [1814]: INFO [fail2ban] Found 66.249.75.108 2016-07-17 09:53:48,827 fail2ban.filter [1814]: INFO [apache-badbots] Found 66.249.75.108 2016-07-17 09:53:48,910 fail2ban.actions [1814]: NOTICE [apache-badbots] 66.249.75.108 already banned 2016-07-17 09:53:49,126 fail2ban.filter [1814]: INFO [apache-fakegooglebot] Ignore 66.249.75.108 by command 2016-07-17 09:53:53,548 fail2ban.filter [1814]: INFO [apache-badbots] Found 66.249.69.136 /snip

Ive been playing with the regex.... but nothing yet

[leoaloha]

also I have maxretry set to 1 and time as default

What does this mean? ---Also, it's highly possible that fail2ban-subnets is just doing one iteration because other offenders were not classified in common subnets.

[XaF]

This means that we must have at least 5 (default min_ips parameter) different IPs that have been banned in a given IP subnet for the subnet to be banned. The total cumulated number of bans of these IPs must also be greater than maxretry. It thus means that if you have less than min_ips IPs in the subnet, or less than maxretry bans for this subnet, the rule for the subnet will not be applied!

[leoaloha]

here is a grepped list of ipaddress from my fail2ban log. You will see many occurrences of 5 or more ip's for subnettable banning. It is quite possible I am using the subnet program incorrectly. I assume once an hour, fail2ban-subnets.py parses the fail2ban.log file and looks for ip's to subnet and increases the range of the subnet as more are found ( depending on the minip value) and inserting them into the iptables as needed

iplist.txt

[XaF]

Could you provide the fail2ban-subnets.cover file generated from the execution of fail2ban-subnets ?

[leoaloha]

changed the name to *.cover.txt fail2ban-subnets.cover.txt

[XaF]

From the following lines:

25687:         m = lineregex.match(l)
       #        print(l," ",m)
25687:         if not m:
25687:             continue

We can see that the regex seems to be the problem. Could you put back the original regex and run it again ? If you see the same pattern, then, would you be able to provide a full log file on which I could try the regex ? (From the example you provided earlier I didn't see any problem as they look like mines, but mines don't seem to have regex problems...)

Also, could you tell me the version of Python you're using ?

[leoaloha]

Ok, the regex back to original and reran the program

fail2ban-subnets.cover.txt

[leoaloha]

Python 2.7.12

[XaF]

It seems you still kept the regex you changed. Even if it shouldn't cause any problem here. Would you mind trying to run your trace on the original file that you can find here so I'm sure we're doing the same?

Also, could you provide the fail2ban.log file you're using? That way I'll be able to verify the regex matching on my side.

[leoaloha]

fail2ban-subnets.cover.txt

[Uploading fail2ban.log.1.txt…]()

[leoaloha]

i had to use a smaller log file it has quite a few banned IPs that are subnetable i also reran the NEW fail2ban-subnets.py file against it

fail2ban.log.txt fail2ban-subnets.cover.txt