Content scrapers log

[Nirjonadda]

sisfeed.com using cloudflare but I have found their own IP (sisfeed.com/IP: 176.99.4.10/11), Does this log help?

176.99.4.10 - - [17/Apr/2017:15:49:56 +0600] "GET / HTTP/1.1" 200 229413 "" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
176.99.4.11 - - [17/Apr/2017:15:49:58 +0600] "GET /cometchat/cometchatcss.php HTTP/1.1" 200 55650 "http://www.mysite.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
176.99.4.11 - - [17/Apr/2017:15:49:58 +0600] "GET /cometchat/cometchatjs.php HTTP/1.1" 200 728214 "http://www.mysite.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"

[mitchellkrogza]

Please send me the logfiles via email without modifying them to remove your domain name, please send at least 10 lines if possible.

[Nirjonadda]

@mitchellkrogza I have sending Log from /var/log/nginx/access.log

[mitchellkrogza]

Thanks got your logs. Wait for the update this week which will allow you use a new /bots.d/blacklist-ips.conf include file to block their IP addresses. That's unfortunately the only way you will block them as they use a very standard User-Agent string and they also send a blank referrer string.

[Nirjonadda]

@mitchellkrogza Thanks i will wait for update but for now I have used deny their IP by .htaccess. I have found now their are using multiple sites with IP.

IP deny by .htaccess their IP

order allow,deny
deny from 5.45.70.31
deny from 5.45.70
deny from 176.99.4.11
deny from 176.99.4.10
deny from 37.1.205.200
deny from 185.162.9.147
deny from 37.1.202.109
deny from 37.1.201.95
deny from 138.201.19.77
deny from 2a01:04f8:0171:2a4c:0000:0000:0000:0002
allow from all

[mitchellkrogza]

Great, new update will allow you to add that into your own custom include and they will not be deleted after update. See https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/Engintron_for_cPanel_WHM_Configuration_Example/usr/sbin/fix-engintron for a quick fix after Enginetron deletes your files.

[Nirjonadda]

@mitchellkrogza Its Impossible manually Identify their IP and block it to prevent. When they are also using CloudFlare.

[mitchellkrogza]

A lot of those IP's in your logs are already generating lots of 403 forbidden errors. Why are you not using Fail2Ban to detect and block them?

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/tree/master/Fail2Ban

Is your site running Wordpress?

My nginx repeat offender filter and action handle this very nicely and you can customise the jail to block them out longer than I do.

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/Fail2Ban/filter.d/nginxrepeatoffender.conf

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/Fail2Ban/action.d/nginxrepeatoffender.conf

[nginxrepeatoffender]
enabled = true
logpath = %(nginx_access_log)s
filter = nginxrepeatoffender
banaction = nginxrepeatoffender
bantime  = -1   ; forever
findtime = 604800   ; 1 week
maxretry = 10

[Nirjonadda]

@mitchellkrogza No, I am using xenForo. Fail2Ban are not come from nginx-ultimate-bad-bot-blocker by default setting? If no so how to install Fail2Ban in cPanel with Enginetron and nginx-ultimate-bad-bot-blocker support?

[mitchellkrogza]

Not sure. What is your operating system? Ubuntu or CentOS ????

[Nirjonadda]

cPanel & WHM 64.0 (build 12) with CENTOS 7.3 x86_64 kvm - CentOS Linux release 7.3.1611 (Core), Also i am using ConfigServer provides WHM plugin CSF.

[Nirjonadda]

@mitchellkrogza I have Installed Fail2ban on CentOS 7, Please let me know that what next now ?

[root@na ~]# systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-04-17 17:36:36 +06; 9s ago
     Docs: man:fail2ban(1)
  Process: 2194 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
 Main PID: 2197 (fail2ban-server)
   CGroup: /system.slice/fail2ban.service
           └─2197 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b

Apr 17 17:36:36 na.mysite.com systemd[1]: Starting Fail2Ban Service...
Apr 17 17:36:36 na.mysite.com fail2ban-client[2194]: 2017-04-17 17:36:36,368 fail2ban.server         [2195]: INFO    Starting Fail2ban v0.9.6
Apr 17 17:36:36 na.mysite.com fail2ban-client[2194]: 2017-04-17 17:36:36,368 fail2ban.server         [2195]: INFO    Starting in daemon mode
Apr 17 17:36:36 na.mysite.com systemd[1]: Started Fail2Ban Service.
[root@na ~]#

[mitchellkrogza]

On CentOS 7

sudo yum install epel-release say Y to all

sudo yum install fail2ban say Y to all

cd /etc/fail2ban/action.d

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/Fail2Ban/action.d/nginxrepeatoffender.conf -O nginxrepeatoffender.conf

cd /etc/fail2ban/filter.d

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/Fail2Ban/filter.d/nginxrepeatoffender.conf -O nginxrepeatoffender.conf

cd /etc/failban

sudo nano jail.local

paste these contents into jail.local and save the file CTRL+X > Y > ENTER this uses nano text editor

# Local Jail.conf File
ignoreip = 127.0.0.1/8
ignorecommand =
bantime  = 600
findtime  = 600
maxretry = 6
backend = auto
usedns = yes
logencoding = auto
enabled = false
filter = %(__name__)s
destemail = you@youremail.com
sender = fail2ban@yourdomain.com
sendername = Fail2Ban
mta = mail
# or mta = sendmail
protocol = tcp
chain = INPUT
port = 0:65535
banaction = iptables-multiport
action = %(action_mwl)s

# JAILS
[nginxrepeatoffender]
enabled = true
logpath = %(nginx_access_log)s
filter = nginxrepeatoffender
banaction = nginxrepeatoffender
bantime  = -1   ; Forever
findtime = 604800   ; 1 week
maxretry = 6

sudo systemctl enable fail2ban

sudo service fail2ban restart

systemctl status fail2ban to make sure it's running.

YOU MUST DO Further customizing of your Fail2ban jail.local file to whitelist your own IP's, set email notifications, add additional jails etc .... you have to find the examples for customizing that in the main jail.conf.

Never modify jail.conf only add required settings and jails into jail.local that you will have to learn all of that from Fail2Ban documentation yourself.

This is only enough instructions to JUST get my nginxrepeatoffender jail working. Change email addresses in above jail.local example to your own and WHITELIST your own ips (all on the same line seperated by a space) ... the rest you will have to learn all of that from Fail2Ban documentation yourself on additional jails etc.

I have just done and tested the instructions above on CentOS7, works !!!

Default clean install of CentOS7 already has iptables installed so it should work 100%. you can check the jail is added to your iptables by running

sudo iptables -S

You will see something similar to this

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N OUTPUT_direct
-N f2b-nginxrepeatoffender
-A INPUT -p tcp -j f2b-nginxrepeatoffender
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A f2b-nginxrepeatoffender -j RETURN

Once again further customization of iptables is all up to you, out of scope for my support for the blocker.

[Nirjonadda]

@mitchellkrogza Where i can find out Fail2ban jail.local file? /etc/fail2ban/ does not have jail.local file

[mitchellkrogza]

Read my instructions above. They are step by step and very simple, I just did this on a fresh CentOS7 to test before posting the instructions here. You have to make jail.local it's in my instructions above this.

[Nirjonadda]

@mitchellkrogza fail2ban restart are not work.

[root@na fail2ban]# systemctl enable fail2ban
[root@na fail2ban]# service fail2ban restart
Redirecting to /bin/systemctl restart  fail2ban.service
Job for fail2ban.service failed because the control process exited with error code. See "systemctl status fail2ban.service" and "journalctl -xe" for details.
[root@na fail2ban]#

[Nirjonadda]

@mitchellkrogza fail2ban Does not start with jail.local files, Now delete jail.local files then fail2ban are working. Please let me know that fix.

[mitchellkrogza]

DOES work with jail.local that is how Fail2ban is designed, go read their docs. I did the steps above on CentOS7 with a fresh install of Fail2Ban and Nginx and it works first time 100%. You have made a typo in your jail.local probably. I have done the above detailed instructions 3 times in a row now and they work each time. I always test thoroughly before I publish instructions.

PLEASE USE NEW INSTRUCTIONS BELOW ..... just use that in your jail.local, nothing else. CentOS has problems suddenly with jail.local 😠

Stop Failban using

sudo fail2ban-client -vvv -x stop

then start is using

sudo fail2ban-client -vvv -x start

And it will lead you to where you have made an error.

Trust me I run Fail2Ban on about 15 servers they all work the same with jail.local and that's on Ubuntu, CentOS, LinuxMint and Debian servers .... works the same on all of them !!!!

Once you find your error

Then sudo service fail2ban restart

[mitchellkrogza]

Follow THIS step by step, ignore earlier jail.local, CentOS (like CPanel people) have managed to MESS up Fail2Ban from the way it is designed to work. 😠 😡

sudo yum install epel-release say Y to all

sudo yum install fail2ban say Y to all

cd /etc/fail2ban/action.d

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_fail2ban_addon/action.d/nginxrepeatoffender.conf -O nginxrepeatoffender.conf

cd /etc/fail2ban/filter.d

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_fail2ban_addon/filter.d/nginxrepeatoffender.conf -O nginxrepeatoffender.conf

cd /etc/failban

sudo nano jail.local

paste these contents into jail.local and save the file CTRL+X > Y > ENTER this uses nano text editor DON'T USE THE ABOVE jail.local I gave you. Something stupid CentOS has done 😡 , it worked initially but then also crashed on me. Hate it when distro's mess with things and don't follow standards.

# Local Jail.conf File
# JAILS
[nginxrepeatoffender]
enabled = true
logpath = %(nginx_access_log)s
filter = nginxrepeatoffender
banaction = nginxrepeatoffender
bantime  = -1   ; Forever
findtime = 604800   ; 1 week
maxretry = 6

sudo systemctl enable fail2ban

sudo service fail2ban restart

systemctl status fail2ban to make sure it's running.

Then edit the email address notifications and ignoreip sections in jail.conf which goes AGAINST they way Fail2Ban is designed but the CentOS7 repo maintainers are 😜 😡 and have managed to mess up the way Fail2Ban is designed to work.

Would love to hear comments from https://github.com/CentOS and centos-devel@centos.org @arrfab @athmane @cgwalters @jperrin @kbsingh @skottler

[mitchellkrogza]

Have you got it working now ??

[Nirjonadda]

@mitchellkrogza Yes now working with jail.local file with this code, Please let me know that where we can edit the email address notifications via jail.conf files?

# Sender email address used solely for some actions
sender = root@localhost

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

[root@na ~]# fail2ban-client status
Status
|- Number of jail:      1
`- Jail list:   nginxrepeatoffender
[root@na ~]#

[mitchellkrogza]

🙏 thank heavens !!! So glad you got it up and running. Persistence pays off. Hope all of this info helps others using Engintron.

Just add

destemail = you@youremail.com and sendername = Fail2Ban

into jail.conf and not jail.local

[mitchellkrogza]

I am going to log an issue on CentOS repo's as they have failed to play nicely with how jail.local files work, very annoying indeed. But glad we got it figured out nonetheless.

[Nirjonadda]

I added via jail.conf, Please let me know that does this correct?

# Sender email address used solely for some actions
sender = root@localhost
destemail = you@youremail.com
sendername = Fail2Ban

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

[mitchellkrogza]

Backup your existing jail.conf

sudo mv /etc/fail2ban/jail.conf /etc/fail2ban/jail.backupconf

then make a new one with

sudo nano /etc/fail2ban/jail.conf

and paste the contents below, just change the destemail to your own email, then restart fail2ban and see if you start getting email notices.

#
# WARNING: heavily refactored in 0.9.0 release.  Please review and
#          customize settings for your setup.
#
# Changes:  in most of the cases you should not modify this
#           file, but provide customizations in jail.local file,
#           or separate .conf files under jail.d/ directory, e.g.:
#
# HOW TO ACTIVATE JAILS:
#
# YOU SHOULD NOT MODIFY THIS FILE.
#
# It will probably be overwritten or improved in a distribution update.
#
# Provide customizations in a jail.local file or a jail.d/customisation.local.
# For example to change the default bantime for all jails and to enable the
# ssh-iptables jail the following (uncommented) would appear in the .local file.
# See man 5 jail.conf for details.
#
# [DEFAULT]
# bantime = 3600
#
# [sshd]
# enabled = true
#
# See jail.conf(5) man page for more information

# Comments: use '#' for comment lines and ';' (following a space) for inline comments

[INCLUDES]

#before = paths-distro.conf
before = paths-fedora.conf

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space (and/or comma) separator.
ignoreip = 127.0.0.1/8

# External command that will take an tagged arguments to ignore, e.g. <ip>,
# and return true if the IP is to be ignored. False otherwise.
#
# ignorecommand = /path/to/command <ip>
ignorecommand =

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 5

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling", "systemd" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#              If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#              If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# systemd:   uses systemd python library to access the systemd journal.
#              Specifying "logpath" is not valid for this backend.
#              See "journalmatch" in the jails associated filter config
# auto:      will try to use the following backends, in order:
#              pyinotify, gamin, polling.
#
# Note: if systemd backend is chosen as the default but you enable a jail
#       for which logs are present only in its own log files, specify some other
#       backend for that jail (e.g. polling) and provide empty value for
#       journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200
backend = auto

# "usedns" specifies if jails should trust hostnames in logs,
#   warn when DNS lookups are performed, or ignore all hostnames in logs
#
# yes:   if a hostname is encountered, a DNS lookup will be performed.
# warn:  if a hostname is encountered, a DNS lookup will be performed,
#        but it will be logged as a warning.
# no:    if a hostname is encountered, will not be used for banning,
#        but it will be logged as info.
# raw:   use raw value (no hostname), allow use it for no-host filters/actions (example user)
usedns = warn

# "logencoding" specifies the encoding of the log files handled by the jail
#   This is used to decode the lines from the log file.
#   Typical examples:  "ascii", "utf-8"
#
#   auto:   will use the system locale setting
logencoding = auto

# "enabled" enables the jails.
#  By default all jails are disabled, and it should stay this way.
#  Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true:  jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false

# "filter" defines the filter to use by the jail.
#  By default jails have names matching their filter name
#
filter = %(__name__)s

#
# ACTIONS
#

# Some options used for actions

# Destination email address used solely for the interpolations in
# jail.{conf,local,d/*} configuration files.
destemail = mitchellkrog@gmail.com

# Sender email address used solely for some actions
sender = root@localhost
sendername = Fail2Ban

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

# Default protocol
protocol = tcp

# Specify chain where jumps would need to be added in iptables-* actions
chain = INPUT

# Ports to be banned
# Usually should be overridden in a particular jail
port = 0:65535

# Format of user-agent https://tools.ietf.org/html/rfc7231#section-5.5.3
fail2ban_agent = Fail2Ban/%(fail2ban_version)s

#
# Action shortcuts. To be used to define action parameter

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
banaction = iptables-multiport
#banaction_allports = iptables-allports

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(mta)s-whois[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# See the IMPORTANT note in action.d/xarf-login-attack for when to use this action
#
# ban & send a xarf e-mail to abuse contact of IP address and include relevant log lines
# to the destemail.
action_xarf = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath=%(logpath)s, port="%(port)s"]

# ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
# to the destemail.
action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
                %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# Report block via blocklist.de fail2ban reporting service API
# 
# See the IMPORTANT note in action.d/blocklist_de.conf for when to
# use this action. Create a file jail.d/blocklist_de.local containing
# [Init]
# blocklist_de_apikey = {api key from registration]
#
action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]

# Report ban via badips.com, and use as blacklist
#
# See BadIPsAction docstring in config/action.d/badips.py for
# documentation for this action.
#
# NOTE: This action relies on banaction being present on start and therefore
# should be last action defined for a jail.
#
action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"]
#
# Report ban via badips.com (uses action.d/badips.conf for reporting only)
#
action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]

# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
# globally (section [DEFAULT]) or per specific section
action = %(action_mwl)s

#
# JAILS
#

#
# SSH servers
#

[sshd]

port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

[sshd-ddos]
# This jail corresponds to the standard configuration in Fail2ban.
# The mail-whois action send a notification e-mail with a whois request
# in the body.
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

[dropbear]

port     = ssh
logpath  = %(dropbear_log)s
backend  = %(dropbear_backend)s

[selinux-ssh]

port     = ssh
logpath  = %(auditd_log)s

#
# HTTP servers
#

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s

[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 172800
maxretry = 1

[apache-noscript]

port     = http,https
logpath  = %(apache_error_log)s

[apache-overflows]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-nohome]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-botsearch]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-fakegooglebot]

port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>

[apache-modsecurity]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-shellshock]

port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

[openhab-auth]

filter = openhab
action = iptables-allports[name=NoAuthFailures]
logpath = /opt/openhab/logs/request.log

[nginx-http-auth]

port    = http,https
logpath = %(nginx_error_log)s

# To use 'nginx-limit-req' jail you should have `ngx_http_limit_req_module` 
# and define `limit_req` and `limit_req_zone` as described in nginx documentation
# http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
# or for example see in 'config/filter.d/nginx-limit-req.conf'
[nginx-limit-req]
port    = http,https
logpath = %(nginx_error_log)s

[nginx-botsearch]

port     = http,https
logpath  = %(nginx_error_log)s
maxretry = 2

# Ban attackers that try to use PHP's URL-fopen() functionality
# through GET/POST variables. - Experimental, with more than a year
# of usage in production environments.

[php-url-fopen]

port    = http,https
logpath = %(nginx_access_log)s
          %(apache_access_log)s

[suhosin]

port    = http,https
logpath = %(suhosin_log)s

[lighttpd-auth]
# Same as above for Apache's mod_auth
# It catches wrong authentifications
port    = http,https
logpath = %(lighttpd_error_log)s

#
# Webmail and groupware servers
#

[roundcube-auth]

port     = http,https
logpath  = %(roundcube_errors_log)s

[openwebmail]

port     = http,https
logpath  = /var/log/openwebmail.log

[horde]

port     = http,https
logpath  = /var/log/horde/horde.log

[groupoffice]

port     = http,https
logpath  = /home/groupoffice/log/info.log

[sogo-auth]
# Monitor SOGo groupware server
# without proxy this would be:
# port    = 20000
port     = http,https
logpath  = /var/log/sogo/sogo.log

[tine20]

logpath  = /var/log/tine20/tine20.log
port     = http,https

#
# Web Applications
#
#

[drupal-auth]

port     = http,https
logpath  = %(syslog_daemon)s
backend  = %(syslog_backend)s

[guacamole]

port     = http,https
logpath  = /var/log/tomcat*/catalina.out

[monit]
#Ban clients brute-forcing the monit gui login
port = 2812
logpath  = /var/log/monit

[webmin-auth]

port    = 10000
logpath = %(syslog_authpriv)s
backend = %(syslog_backend)s

[froxlor-auth]

port    = http,https
logpath  = %(syslog_authpriv)s
backend  = %(syslog_backend)s

#
# HTTP Proxy servers
#
#

[squid]

port     =  80,443,3128,8080
logpath = /var/log/squid/access.log

[3proxy]

port    = 3128
logpath = /var/log/3proxy.log

#
# FTP servers
#

[proftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(proftpd_log)s
backend  = %(proftpd_backend)s

[pure-ftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(pureftpd_log)s
backend  = %(pureftpd_backend)s

[gssftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(syslog_daemon)s
backend  = %(syslog_backend)s

[wuftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(wuftpd_log)s
backend  = %(wuftpd_backend)s

[vsftpd]
# or overwrite it in jails.local to be
# logpath = %(syslog_authpriv)s
# if you want to rely on PAM failed login attempts
# vsftpd's failregex should match both of those formats
port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(vsftpd_log)s

#
# Mail servers
#

# ASSP SMTP Proxy Jail
[assp]

port     = smtp,465,submission
logpath  = /root/path/to/assp/logs/maillog.txt

[courier-smtp]

port     = smtp,465,submission
logpath  = %(syslog_mail)s
backend  = %(syslog_backend)s

[postfix]

port     = smtp,465,submission
logpath  = %(postfix_log)s
backend  = %(postfix_backend)s

[postfix-rbl]

port     = smtp,465,submission
logpath  = %(postfix_log)s
backend  = %(postfix_backend)s
maxretry = 1

[sendmail-auth]

port    = submission,465,smtp
logpath = %(syslog_mail)s
backend = %(syslog_backend)s

[sendmail-reject]

port     = smtp,465,submission
logpath  = %(syslog_mail)s
backend  = %(syslog_backend)s

[qmail-rbl]

filter  = qmail
port    = smtp,465,submission
logpath = /service/qmail/log/main/current

# dovecot defaults to logging to the mail syslog facility
# but can be set by syslog_facility in the dovecot configuration.
[dovecot]

port    = pop3,pop3s,imap,imaps,submission,465,sieve
logpath = %(dovecot_log)s
backend = %(dovecot_backend)s

[sieve]

port   = smtp,465,submission
logpath = %(dovecot_log)s
backend = %(dovecot_backend)s

[solid-pop3d]

port    = pop3,pop3s
logpath = %(solidpop3d_log)s

[exim]

port   = smtp,465,submission
logpath = %(exim_main_log)s

[exim-spam]

port   = smtp,465,submission
logpath = %(exim_main_log)s

[kerio]

port    = imap,smtp,imaps,465
logpath = /opt/kerio/mailserver/store/logs/security.log

#
# Mail servers authenticators: might be used for smtp,ftp,imap servers, so
# all relevant ports get banned
#

[courier-auth]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
logpath  = %(syslog_mail)s
backend  = %(syslog_backend)s

[postfix-sasl]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
# You might consider monitoring /var/log/mail.warn instead if you are
# running postfix since it would provide the same log lines at the
# "warn" level but overall at the smaller filesize.
logpath  = %(postfix_log)s
backend  = %(postfix_backend)s

[perdition]

port   = imap3,imaps,pop3,pop3s
logpath = %(syslog_mail)s
backend = %(syslog_backend)s

[squirrelmail]

port = smtp,465,submission,imap2,imap3,imaps,pop3,pop3s,http,https,socks
logpath = /var/lib/squirrelmail/prefs/squirrelmail_access_log

[cyrus-imap]

port   = imap3,imaps
logpath = %(syslog_mail)s
backend = %(syslog_backend)s

[uwimap-auth]

port   = imap3,imaps
logpath = %(syslog_mail)s
backend = %(syslog_backend)s

#
#
# DNS servers
#

# !!! WARNING !!!
#   Since UDP is connection-less protocol, spoofing of IP and imitation
#   of illegal actions is way too simple.  Thus enabling of this filter
#   might provide an easy way for implementing a DoS against a chosen
#   victim. See
#    http://nion.modprobe.de/blog/archives/690-fail2ban-+-dns-fail.html
#   Please DO NOT USE this jail unless you know what you are doing.
#
# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks UDP traffic for DNS requests.
# [named-refused-udp]
#
# filter   = named-refused
# port     = domain,953
# protocol = udp
# logpath  = /var/log/named/security.log

# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks TCP traffic for DNS requests.

[named-refused]

port     = domain,953
logpath  = /var/log/named/security.log

[nsd]

port     = 53
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
logpath = /var/log/nsd.log

#
# Miscellaneous
#

[asterisk]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/asterisk/messages
maxretry = 10

[freeswitch]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/freeswitch.log
maxretry = 10

# To log wrong MySQL access attempts add to /etc/my.cnf in [mysqld] or
# equivalent section:
# log-warning = 2
#
# for syslog (daemon facility)
# [mysqld_safe]
# syslog
#
# for own logfile
# [mysqld]
# log-error=/var/log/mysqld.log
[mysqld-auth]

port     = 3306
logpath  = %(mysql_log)s
backend  = %(mysql_backend)s

# Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf')
[mongodb-auth]
# change port when running with "--shardsvr" or "--configsvr" runtime operation
port     = 27017
logpath  = /var/log/mongodb/mongodb.log

# Jail for more extended banning of persistent abusers
# !!! WARNINGS !!!
# 1. Make sure that your loglevel specified in fail2ban.conf/.local
#    is not at DEBUG level -- which might then cause fail2ban to fall into
#    an infinite loop constantly feeding itself with non-informative lines
# 2. Increase dbpurgeage defined in fail2ban.conf to e.g. 648000 (7.5 days)
#    to maintain entries for failed logins for sufficient amount of time
[recidive]

logpath  = /var/log/fail2ban.log
banaction = %(banaction_allports)s
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day

# Generic filter for PAM. Has to be used with action which bans all
# ports such as iptables-allports, shorewall

[pam-generic]
# pam-generic filter can be customized to monitor specific subset of 'tty's
banaction = %(banaction_allports)s
logpath  = %(syslog_authpriv)s
backend  = %(syslog_backend)s

[xinetd-fail]

banaction = iptables-multiport-log
logpath   = %(syslog_daemon)s
backend   = %(syslog_backend)s
maxretry  = 2

# stunnel - need to set port for this
[stunnel]

logpath = /var/log/stunnel4/stunnel.log

[ejabberd-auth]

port    = 5222
logpath = /var/log/ejabberd/ejabberd.log

[counter-strike]

logpath = /opt/cstrike/logs/L[0-9]*.log
# Firewall: http://www.cstrike-planet.com/faq/6
tcpport = 27030,27031,27032,27033,27034,27035,27036,27037,27038,27039
udpport = 1200,27000,27001,27002,27003,27004,27005,27006,27007,27008,27009,27010,27011,27012,27013,27014,27015
action  = %(banaction)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]

# consider low maxretry and a long bantime
# nobody except your own Nagios server should ever probe nrpe
[nagios]

logpath  = %(syslog_daemon)s     ; nrpe.cfg may define a different log_facility
backend  = %(syslog_backend)s
maxretry = 1

[oracleims]
# see "oracleims" filter file for configuration requirement for Oracle IMS v6 and above
logpath = /opt/sun/comms/messaging64/log/mail.log_current
banaction = %(banaction_allports)s

[directadmin]
logpath = /var/log/directadmin/login.log
port = 2222

[portsentry]
logpath  = /var/lib/portsentry/portsentry.history
maxretry = 1

[pass2allow-ftp]
# this pass2allow example allows FTP traffic after successful HTTP authentication
port         = ftp,ftp-data,ftps,ftps-data
# knocking_url variable must be overridden to some secret value in jail.local
knocking_url = /knocking/
filter       = apache-pass[knocking_url="%(knocking_url)s"]
# access log of the website with HTTP auth
logpath      = %(apache_access_log)s
blocktype    = RETURN
returntype   = DROP
bantime      = 3600
maxretry     = 1
findtime     = 1

[murmur]
# AKA mumble-server
port     = 64738
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol=tcp, chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol=udp, chain="%(chain)s", actname=%(banaction)s-udp]
logpath  = /var/log/mumble-server/mumble-server.log

[screensharingd]
# For Mac OS Screen Sharing Service (VNC)
logpath  = /var/log/system.log
logencoding = utf-8

[haproxy-http-auth]
# HAProxy by default doesn't log to file you'll need to set it up to forward
# logs to a syslog server which would then write them to disk.
# See "haproxy-http-auth" filter for a brief cautionary note when setting
# maxretry and findtime.
logpath  = /var/log/haproxy.log

[slapd]
port    = ldap,ldaps
filter  = slapd
logpath = /var/log/slapd.log

[jperrin]

Please note that CentOS has nothing to do with fail2ban.

This is a package from the EPEL (produced by Fedora - https://fedoraproject.org/wiki/EPEL ) repository. Please submit your bug there via bugzilla.

[mitchellkrogza]

@jperrin thanks so much for the clarification. I will log issue there. Thanks for the reply. Have managed to get it working now for my user but it clearly does not work the same way with jail.local files as it does on other distro's. Thanks again.

[Nirjonadda]

@mitchellkrogza Yes, Fail2Ban running does now Fail2Ban to detect IP and block them from Content scrapers? Does work Block Json Scrapers, Block Cookieless Bots Earlier and Detect JavaScript ?

[mitchellkrogza]

@Nirjonadda That single jail I gave you will ONLY pick up 403 and 444 errors from your nginx logs and ban them when they make too many attempts.

Other jails you can look at for Fail2Ban are: (example below from my jail.local) Please DON'T just copy and paste the extra jails into your jail.local the nginx-noscript jail needs to be created first

see: https://ubuntu101.co.za/security/fail2ban/fail2ban-persistent-bans-ubuntu/

also see https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning

Unfortunately you are going to have to learn Fail2Ban and how to configure it and add jails, it's part of server management and you need to learn how it works and how to fix it when you break it.

It's out of scope for what this repository support is for.

#
# Local Jail.conf File
# JAILS
#

#
# SSH servers
#

[ssh]
enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.*
maxretry = 6

[ssh-ddos]
enabled  = true
port     = ssh
filter   = sshd-ddos
logpath  = /var/log/auth.*
maxretry = 6

#
# HTTP servers
#

[nginx-http-auth]
enabled = true
port    = http,https
filter = nginx-http-auth
logpath = %(nginx_access_log)s
maxretry = 3

[nginx-botsearch]
enabled = true
port     = http,https
filter = nginx-botsearch
logpath = %(nginx_access_log)s
maxretry = 6

[nginx-noscript]
enabled = false
port     = http,https
filter = nginx-noscript
logpath = %(nginx_access_log)s
maxretry = 6

[nginx-nohome]
enabled  = true
port     = http,https
filter   = nginx-nohome
logpath = %(nginx_access_log)s
maxretry = 6

[nginx-limit-req]
enabled = true
port    = http,https
filter = nginx-limit-req
logpath = %(nginx_error_log)s
maxretry = 1

[nginxrepeatoffender]
enabled = true
logpath = %(nginx_access_log)s
filter = nginxrepeatoffender
banaction = nginxrepeatoffender
bantime  = -1   ; 1 day
findtime = 604800   ; 1 week
maxretry = 10

[blacklist]
enabled = true
logpath  = /var/log/fail2ban.*
filter = blacklist
banaction = blacklist
#action = %(action_mybadips)s
bantime  = 31536000   ; 1 year
findtime = 31536000   ; 1 year
maxretry = 6

[Nirjonadda]

@mitchellkrogza I am still are not getting any email from Fail2Ban. Also does not see the Content scrapers are blocking. Do you need my Log from /var/log/nginx/access.log?

[mitchellkrogza]

Have you tested your server can even send email ???? Try this from command line.

mail -s "Test - EMAIL" you@youremailaddress.com < /dev/null

[mitchellkrogza]

You can check if Fail2Ban is blocking things by running

sudo iptables -S

You will see things like this near the bottom

-A f2b-nginxrepeatoffender -s 95.59.143.166/32 -j DROP
-A f2b-nginxrepeatoffender -s 95.142.66.23/32 -j DROP
-A f2b-nginxrepeatoffender -s 94.102.14.252/32 -j DROP

[Nirjonadda]

@mitchellkrogza Email test are OK but why not get email? Does its block also IPv6?

[mitchellkrogza]

No Fail2Ban does not block IPV6 yet …. being implemented in Fail2Ban 0.10.x branch (beta)

From: Md Tajirul Islam notifications@github.com notifications@github.com Reply: mitchellkrogza/nginx-ultimate-bad-bot-blocker reply@reply.github.com reply@reply.github.com Date: 22 April 2017 at 10:51:39 AM To: mitchellkrogza/nginx-ultimate-bad-bot-blocker nginx-ultimate-bad-bot-blocker@noreply.github.com nginx-ultimate-bad-bot-blocker@noreply.github.com Cc: Mitchell Krog mitchellkrog@gmail.com mitchellkrog@gmail.com, Mention mention@noreply.github.com mention@noreply.github.com Subject: Re: [mitchellkrogza/nginx-ultimate-bad-bot-blocker] Content scrapers log (#26)

@mitchellkrogza https://github.com/mitchellkrogza Email test are OK but why not get email? Does its block also IPv6?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/26#issuecomment-296358730, or mute the thread https://github.com/notifications/unsubscribe-auth/AJgARXUSbjvo-I3VaXcgSm1mQoCBN6-jks5ryb-bgaJpZM4M_CeZ .

[mitchellkrogza]

See - https://github.com/fail2ban/fail2ban/issues/1123

[Nirjonadda]

OK, Waiting for Fail2Ban 0.10.x branch stable versions. @mitchellkrogza How to check that what fail2ban version I am used?

[mitchellkrogza]

We all are, I think we will see stable branch later this year.

[mitchellkrogza]

@Nirjonadda links in the steps to install the add-on have been fixed