Content scrapers not getting blocked.

RealSuprim commented 4 years ago

Hi, I am using sucuri firewall and I was having issue with content scraper, I looked at my log and blocked some ip which were making a lot of request and it did stop the scraper for a day and next day it started to scrape again. Now even blocking ip is not stopping them.

I found this ultimate bad bot blocker project and installed it few days ago, and the scraper has not stopped. How do I block them?

Thanks

mitchellkrogza commented 4 years ago

Post some log samples of what their bot is doing. Have you added their IP's to https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/bots.d/blacklist-ips.conf ??

GitHub
mitchellkrogza/nginx-ultimate-bad-bot-blocker
Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...

RealSuprim commented 4 years ago

Yes I have added the ip in blacklist-ips.conf .

So I just uploaded new article which has already been scraped and posted on the scam site. here is all the logs for that link

3.85.230.130 - - [13/Jan/2020:15:09:00 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27995 "-" "Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)"

3.91.30.255 - - [13/Jan/2020:15:09:49 +0000] "HEAD /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.43 Safari/536.11"

3.91.30.255 - - [13/Jan/2020:15:09:50 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27996 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.43 Safari/536.11"

66.131.226.129 - - [13/Jan/2020:15:09:50 +0000] "POST /wp-admin/admin-ajax.php?td_theme_name=Newspaper&v=10.2 HTTP/2.0" 200 1937 "/video/category/premier-league/" "Mozilla/5.0 (Linux; Android 8.1.0; SM-T580) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.116 Safari/537.36"

3.82.48.3 - - [13/Jan/2020:15:09:51 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27997 "-" "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"

75.101.234.61 - - [13/Jan/2020:15:09:51 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27996 "-" "PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)"

111.93.59.130 - - [13/Jan/2020:15:10:28 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27998 "-" "Mozilla/5.0 (Linux; Android 8.1.0; SM-T835 Build/M1AJQ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.85 Safari/537.36"

54.80.233.198 - - [13/Jan/2020:15:10:28 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27994 "-" "Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)"

111.93.59.130 - - [13/Jan/2020:15:10:28 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27998 "-" "Mozilla/5.0 (Linux; Android 8.1.0; SM-T835 Build/M1AJQ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.85 Safari/537.36"

66.248.203.17 - - [13/Jan/2020:06:51:21 +0000] "GET /video/mourinho-refused-to-talk-bad-about-man-united/ HTTP/1.1" 200 28460 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)"

66.249.65.83 - - [13/Jan/2020:15:05:44 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27998 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.65.83 - - [13/Jan/2020:15:05:45 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27999 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.65.83 - - [13/Jan/2020:15:05:52 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27995 "-" "Googlebot-Image/1.0"

66.249.65.85 - - [13/Jan/2020:15:06:21 +0000] "GET /video/players-that-refused-to-leave-their- club/feed/ HTTP/1.1" 200 464 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

185.93.229.17 - - [13/Jan/2020:15:08:19 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27996 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 35.187.132.108 - - [13/Jan/2020:15:11:28 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27997 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)"

35.187.132.104 - - [13/Jan/2020:15:11:29 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27995 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)"

62.84.220.155 - - [13/Jan/2020:15:17:35 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/2.0" 200 27976 "https://mydomain.net/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

148.69.154.61 - - [13/Jan/2020:15:27:08 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/2.0" 200 27979 "https://mydomain.net/" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"

54.39.177.197 - - [13/Jan/2020:15:27:32 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 27995 "-" "Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)"

199.188.201.172 - - [13/Jan/2020:15:30:08 +0000] "GET /video/players-that-refused-to-leave-their-club/ HTTP/2.0" 200 160950 "http://www.bing.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

3.91.30.255 - - [13/Jan/2020:15:09:49 +0000] "HEAD /video/players-that-refused-to-leave-their-club/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.43 Safari/536.11"`

Thanks

mitchellkrogza commented 4 years ago

Edit your bots.d/blacklist-user-agents.conf file and add the following at the bottom (assuming you want to block all of these)

# ------------
# MY BLACKLIST
# ------------

"~*(?:\b)proximic(?:\b)"            3;
"~*(?:\b)comscore\.com(?:\b)"           3;
"~*(?:\b)PhantomJS(?:\b)"           3;
"~*(?:\b)Web-Crawler(?:\b)"         3;
"~*(?:\b)linkfluence\.com(?:\b)"        3;

Add more as you need or as they change their UA name. Make sure to reload nginx after adding this to the include and make SURE you monitor such changes always.

RealSuprim commented 4 years ago

Oh I thought they were some legit crawler. I will keep on monitoring my logs.

Thanks a lot Mitchell.

mitchellkrogza commented 4 years ago

Some of those ARE legitimate, you need to only block the one's actually scraping your content. (proximic, comscore and linkfluence are legit so OK) so just block PhantomJS (very commonly used for scraping and screenshotting web sites)

Other than that ... people stealing your content may use very common user agent names which are unsafe to block, then you need to block by IP or IP Ranges once you identify where they come from first.

RealSuprim commented 4 years ago

Hi Mitchell,

The scrape is still happening I am blocking all the useragents which is not google and bing. I posted a new article and it is scraped.

92.222.100.29 - - [14/Jan/2020:12:50:35 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27989 "https://mydomain.net/feed/" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2973.60 Safari/537.36"

178.200.236.56 - - [14/Jan/2020:12:54:31 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/2.0" 200 27968 "https://mydomain.net/" "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

213.132.147.109 - - [14/Jan/2020:12:55:13 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/2.0" 200 27970 "https://mydomain.net/" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1"

52.202.46.81 - - [14/Jan/2020:12:55:17 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27990 "-" "GumGum-Bot/1.0 (http://gumgum.com; support@gumgum.com)"

103.135.95.162 - - [14/Jan/2020:13:03:28 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/2.0" 200 27970 "https://mydomain.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

54.90.94.8 - - [14/Jan/2020:13:05:31 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27990 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)"

121.6.51.218 - - [14/Jan/2020:13:05:52 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/2.0" 200 27971 "https://mydomain.net/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

3.88.163.208 - - [14/Jan/2020:13:08:16 +0000] "HEAD /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.43 Safari/536.11"

100.26.104.101 - - [14/Jan/2020:13:08:18 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27991 "-" "PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)"

111.93.59.130 - - [14/Jan/2020:13:10:28 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27991 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"

3.216.27.251 - - [14/Jan/2020:13:11:16 +0000] "GET /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 27990 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.43 Safari/536.11"

I added PocketParser to bad user-agent but it is still getting 200 and not 444 errors.

Thanks

RealSuprim commented 4 years ago

Also was just looking at log and found this : 2600:3c03::f03c:91ff:fee7:cbb3 - - [14/Jan/2020:13:26:42 +0000] "HEAD /video/barcelona-sack-ernesto-valverde/ HTTP/1.1" 200 0 "-" "-" It looks like a blank useragent

mitchellkrogza commented 4 years ago

The blocker is not configured properly, it is issuing 200 OK and not 444

mitchellkrogza commented 4 years ago

Post your vhost configuration, note we do not block any blank user agent name, too risky. In this case you have to start blocking them by IP address

RealSuprim commented 4 years ago

It is now blcoking

# ------------
# MY BLACKLIST
# ------------

"~*(?:\b)PocketParser(?:\b)"            3;
"~*(?:\b)getpocket\.com(?:\b)"      3;
"~*(?:\b)VelenPublicWebCrawler(?:\b)"           3;
"~*(?:\b)velen\.io(?:\b)"           3;
"~*(?:\b)Scrapy(?:\b)"          3;
"~*(?:\b)scrapy\.org(?:\b)"         3;
"~*(?:\b)proximic(?:\b)"            3;
"~*(?:\b)comscore\.com(?:\b)"           3;
"~*(?:\b)PhantomJS(?:\b)"           3;
"~*(?:\b)Web-Crawler(?:\b)"         3;
"~*(?:\b)linkfluence\.com(?:\b)"        3;
"~*(?:\b)x22(?:\b)"                 3;
"~*(?:\b){|}(?:\b)"                 3;
"~*(?:\b)mb_ereg_replace(?:\b)"         3;
"~*(?:\b)file_put_contents(?:\b)"           3;

RealSuprim commented 4 years ago

Post your vhost configuration, note we do not block any blank user agent name, too risky. In this case you have to start blocking them by IP address

where is vhost configuration located?

mitchellkrogza commented 4 years ago

Is it blocking them now or not?

Your vhost is your nginx server {} block for the web site

RealSuprim commented 4 years ago

it is blocking that useragent pocketpraser now but not the scraper.

I am really new to nginx server so struggling to find the vhost. is this the vhost?

# Nginx Bad Bot Blocker Includes
    # REPO: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
    ##
    include /etc/nginx/bots.d/ddos.conf; 
    include /etc/nginx/bots.d/blockbots.conf;

    include common/headers-https.conf;
    include common/headers-html.conf;
    include /var/www/mydomain.net/*-nginx.conf;
}

mitchellkrogza commented 4 years ago

@RealSuprim your vhost is everything for your domain between server { and } example:

server {
    listen *:80;
    listen [::]:80;
    server_name mydomain.com www.mydomain.com;

    location / {
        # Block Bad Bots
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;
        }
    access_log /var/log/nginx/access.log redirects;
    error_log /var/log/nginx/error.log;

# END OF HTTP PORT 80 HOST CONFIG - CLOSING BRACE BELOW THIS LINE
}

You will have two blocks one for port 80 and another for port 443 if you are using SSL

RealSuprim commented 4 years ago

Hi Mitchel here is my vhost

# WebinolySSLredirectStart - HTTP to HTTPS Redirect
server {
    listen 80;
    listen [::]:80;
    server_name mydomain.net www.mydomain.net;
    return 301 https://$host$request_uri;
}
# WebinolySSLredirectEnd
# WebinolyNginxServerStart
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name mydomain.net www.mydomain.net;

    # WebinolySSLstart
    ssl_certificate /etc/letsencrypt/live/mydomain.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mydomain.net/privkey.pem;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/mydomain.net/chain.pem;
    # WebinolySSLend    
    access_log /var/log/nginx/mydomain.net.access.log;
    error_log /var/log/nginx/mydomain.net.error.log;

    root /var/www/mydomain.net/htdocs;

    index  index.php index.html index.htm;

    include common/auth.conf;

    # WebinolyCustom
    include common/yoast-sitemap.conf;
    # WebinolyCustomEnd

    include common/php.conf;
    include common/wpcommon.conf;
    include common/locations.conf;
    include common/headers-http.conf;

    ##
    # Nginx Bad Bot Blocker Includes
    # REPO: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
    ##
    include /etc/nginx/bots.d/ddos.conf; 
    include /etc/nginx/bots.d/blockbots.conf;

    include common/headers-https.conf;
    include common/headers-html.conf;
    include /var/www/mydomain.net/*-nginx.conf;
}
# WebinolyNginxServerEnd

mitchellkrogza commented 4 years ago

Can I see the contents of include common/locations.conf;

You say the blocker is blocking pocketparser but not others? Which others?

RealSuprim commented 4 years ago

Hi mitchel

Yes it blocked pocketparser, but the scraper are using blank user agent so I have been blocking all the ip and every Ip was coming from this AS14618 so I blocked all the ip range in this ASN.

I am also getting this request and the referrer is coming from mygoaltv .com and they are also scrapping my content.

96.237.172.202 - - [15/Jan/2020:06:26:45 +0000] "GET /wp-content/uploads/2020/01/test.jpg HTTP/2.0" 200 12855 "http:// mygoaltv .com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1"

How do i block this referrer?

here is my common/locations.conf;

# NGINX CONFIGURATION FOR COMMON LOCATION
# DO NOT MODIFY, ALL CHANGES LOST AFTER UPDATE Webinoly

# Basic locations files
location = /favicon.ico {
    access_log off;
    log_not_found off;
    expires max;
}
location = /robots.txt {
    try_files $uri $uri/ /index.php?$args;
    access_log off;
    log_not_found off;
}

# Cache static files
location ~* \.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf|swf|woff2|m4a|cur|heic|tiff|webm|mp3|aac|webp)$ {
    include common/headers-http.conf;
    include common/headers-https.conf;
    add_header "Access-Control-Allow-Origin" "*";
    access_log off;
    log_not_found off;
    expires max;
}

# Security settings for better privacy
# Deny hidden files
location ~ /\.well-known {
    allow all;
}
location ~ /\. {
    deny all;
    access_log off;
    log_not_found off;
}

# Deny backup extensions & log files
location ~* ^.+\.(old|orig|original|php#|php~|php_bak|save|swo|aspx?|tpl|sh|bash|bak?|cfg|cgi|dll|exe|git|hg|ini|jsp|log|mdb|out|sql|svn|swp|tar|rdf)$ {
    deny all;
    access_log off;
    log_not_found off;
}

# Return 403 forbidden for readme.(txt|html) or license.(txt|html) or example.(txt|html)
location ~*  "/(^$|readme|license|example|README|LEGALNOTICE|INSTALLATION|CHANGELOG)\.(txt|html|md)" {
    return 403;
}

Thanks

mitchellkrogza commented 4 years ago

To block your own referrers add them to https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/bots.d/custom-bad-referrers.conf

so add "~*(?:\b)mygoaltv\.com(?:\b)" 1;

and keep adding new referrers as you find them.

Read the main readme about monitoring your logs with cron

GitHub
mitchellkrogza/nginx-ultimate-bad-bot-blocker
Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...

RealSuprim commented 4 years ago

Hi mitchell,

had added the referrer code wrong, now it should work fine. I have setup the cron for logs.

Thanks

mitchellkrogza commented 4 years ago

Daily monitoring of your logs and continually adding new bad referrers That's how this was built. Feel free to also contribute known bad scrapers and referrers to the project.

RealSuprim commented 4 years ago

Thanks a lot Mitchell. I will add when I find any bad bots.

RealSuprim commented 4 years ago

Hi Mitchell Is there a better way to block ASN ? Currently I have been blocking all the ip ranges which is in ASN.

mitchellkrogza commented 4 years ago

Hi Mitchell Is there a better way to block ASN ? Currently I have been blocking all the ip ranges which is in ASN.

Post your current IP blocks so I can see

RealSuprim commented 4 years ago

Hi here is the list I have blocked.

138.229.98.193 1;
159.20.21.207 1;
2601:646:8a00:bb30:386d:b4a7:e8d5:4f4c 1;
51.159.17.56 1;
103.131.71.114 1;
103.131.71.108 1;
113.185.78.165 1;
171.241.34.127 1;
103.199.40.70 1;
101.99.35.4 1;
87.56.171.155 1;
110.249.202.143 1;
106.38.241.140 1;
2a02:4780:2:2::1a 1;
119.27.168.199 1;
84.22.150.31 1;
5.133.137.203 1;
155.138.151.11  1;
45.17.79.203 1;
2600:1900:2000:1d:400::14 1;
123.24.182.187 1;
14.244.245.110 1;
123.21.86.233 1;
113.161.81.166 1;
27.77.31.219 1;
160.242.137.236 1;
1.0.168.215 1;
110.175.66.172 1;
14.248.65.29 1;
171.255.199.87 1;
117.4.114.171 1;
43.245.216.212 1;
113.172.237.200 1;
14.242.206.28 1;
110.78.148.157 1;
45.121.84.185 1;
116.110.89.45 1;
199.188.201.172 1;
35.233.196.167 1;
110.249.202.47 1;
115.74.84.222 1;
37.252.94.233 1;
37.252.81.94 1;
5.248.255.159 1;
38.143.100.21 1;
185.247.59.251 1;
61.106.82.130 1;
154.160.5.95 1;
46.98.232.199 1;
95.87.250.113 1;
174.91.18.83 1;
42.117.174.117 1;
54.221.124.216 1;
24.66.114.122 1;
42.114.242.62 1;
185.164.138.19 1;
178.22.168.194 1;
102.149.143.222 1;
2001:41d0:2:599c::131f:510a 1;
41.248.188.153 1;
1.54.5.164 1;
42.115.71.177 1;
176.107.188.163 1;
2600:3c03::f03c:91ff:fee7:cbb3 1;
52.206.134.79 1;
77.75.122.202 1;
52.206.134.79 1;
3.85.229.100 1;
206.72.201.66 1;
37.115.118.26 1;
2600:1900:2001:10::19 1;

# ASN14618 amazon
107.21.64.0/18 1;
54.236.128.0/17 1;
174.129.0.0/16 1;
23.133.224.0/24 1;
72.52.62.0/24 1;
54.144.0.0/14 1;
142.0.191.0/24 1;
75.101.128.0/17 1;
107.22.0.0/16 1;
69.72.41.0/24 1;
204.236.192.0/18 1;
50.17.0.0/16 1;
208.75.220.0/22 1;
18.232.0.0/14 1;
45.223.13.0/24 1;
52.0.0.0/15 1;
164.153.101.0/24 1;
137.83.200.0/24 1;
52.128.42.0/24 1;
23.20.0.0/15 1;
192.225.212.0/24 1;
209.94.74.0/24 1;
52.128.40.0/24 1;
98.142.176.0/24 1;
54.174.0.0/15 1;
139.180.244.0/23 1;
198.177.255.0/24 1;
54.224.0.0/15 1;
52.44.0.0/15 1;
2600:1f18:6000::/35 1;
54.89.0.0/16 1;
54.240.32.0/20 1;
192.161.150.0/24 1;
164.153.100.0/24 1;
54.211.0.0/16 1;
85.115.38.0/24 1;
206.190.220.0/24 1;
23.166.224.0/24 1;
64.71.238.0/24 1;
208.80.202.0/23 1;
204.153.219.0/24 1;
72.44.32.0/19 1;
44.192.0.0/11 1;
139.60.1.0/24 1;
35.153.0.0/16 1;
2406:da00:ff00::/48 1;
74.112.132.0/24 1;
137.83.206.0/23 1;
104.171.199.0/24 1;
45.136.240.0/24 1;
68.64.4.0/24 1;
155.46.135.0/24 1;
64.57.9.0/24 1;
107.23.128.0/17 1;
142.0.188.0/24 1;
54.80.0.0/14 1;
54.198.0.0/16 1;
15.177.0.0/21 1;
52.128.41.0/24 1;
99.77.128.0/24 1;
54.172.0.0/15 1;
52.95.245.0/24 1;
142.202.204.0/24 1;
162.247.162.0/24 1;
168.245.155.0/24 1;
52.200.0.0/13 1;
163.253.46.0/24 1;
216.182.238.0/23 1;
52.90.0.0/15 1;
2600:1f18:2000::/35 1;
91.102.56.0/21 1;
204.8.30.0/24 1;
204.8.29.0/24 1;
136.184.224.0/24 1;
18.204.0.0/14 1;
54.88.0.0/16 1;
206.130.43.0/24 1;
192.225.218.0/24 1;
192.92.97.0/24 1;
69.72.43.0/24 1;
50.19.0.0/17 1;
139.180.17.0/24 1;
199.30.176.0/24 1;
142.202.42.0/24 1;
54.236.64.0/18 1;
208.93.103.0/24 1;
139.60.0.0/24 1;
168.149.241.0/24 1;
168.151.30.0/24 1;
3.208.0.0/12 1;
52.72.0.0/15 1;
54.221.0.0/16 1;
130.137.137.0/24 1;
64.57.15.0/24 1;
205.220.189.0/24 1;
99.77.129.0/24 1;
34.224.0.0/12 1;
54.166.0.0/15 1;
139.60.3.0/24 1;
23.22.0.0/15 1;
50.16.0.0/16 1;
207.64.134.0/23 1;
192.111.4.0/24 1;
131.226.191.0/24 1;
198.17.127.0/24 1;
173.213.62.0/24 1;
208.71.23.0/24 1;
76.76.17.0/24 1;
137.83.194.0/24 1;
130.137.89.0/24 1;
136.184.225.0/24 1;
2600:1f18:4000::/35 1;
130.137.82.0/24 1;
54.92.128.0/17 1;
208.78.6.0/24 1;
54.240.8.0/21 1;
199.91.149.0/24 1;
69.59.249.0/24 1;
54.208.0.0/15 1;
173.195.208.0/24 1;
204.8.28.0/24 1;
184.73.0.0/16 1;
199.47.128.0/24 1;
74.112.133.0/24 1;
66.59.62.0/24 1;
192.159.123.0/24 1;
107.21.0.0/18 1;
54.240.48.0/23 1;
54.84.0.0/15 1;
54.237.0.0/16 1;
54.236.0.0/18 1;
184.72.128.0/17 1;
64.238.3.0/24 1;
52.128.43.0/24 1;
208.78.7.0/24 1;
184.72.96.0/19 1;
208.78.4.0/23 1;
192.149.210.0/24 1;
192.161.149.0/24 1;
23.136.176.0/24 1;
100.24.0.0/13 1;
3.80.0.0/12 1;
158.247.16.0/20 1;
147.160.167.0/24 1;
54.240.30.0/23 1;
216.115.22.0/24 1;
162.215.224.0/23 1;
208.88.208.0/24 1;
54.90.0.0/15 1;
54.87.0.0/16 1;
173.213.63.0/24 1;
54.196.0.0/15 1;
107.21.128.0/17 1;
54.242.0.0/15 1;
69.72.40.0/24 1;
192.225.223.0/24 1;
192.161.148.0/24 1;
89.251.12.0/24 1;
168.149.240.0/24 1;
54.204.0.0/15 1;
69.2.101.0/24 1;
204.75.189.0/24 1;
54.234.0.0/15 1;
148.5.72.0/24 1;
64.45.128.0/24 1;
52.2.0.0/15 1;
130.50.125.0/24 1;
52.86.0.0/15 1;
3.224.0.0/12 1;
15.193.6.0/24 1;
198.178.114.0/24 1;
72.13.123.0/24 1;
54.156.0.0/14 1;
137.83.201.0/24 1;
99.77.191.0/24 1;
198.176.127.0/24 1;
216.182.224.0/21 1;
2600:1f18::/35 1;
204.236.224.0/19 1;
216.182.232.0/22 1;
107.20.0.0/16 1;
64.66.45.0/24 1;
161.38.192.0/22 1;
66.59.61.0/24 1;
199.19.192.0/24 1;
54.210.0.0/16 1;
50.19.128.0/17 1;
74.122.241.0/24 1;
205.157.216.0/24 1;
34.192.0.0/12 1;
76.223.191.0/24 1;
198.136.165.0/24 1;
192.146.118.0/24 1;
198.178.115.0/24 1;
142.54.41.0/24 1;
69.64.150.0/24 1;
69.72.42.0/24 1;
52.20.0.0/14 1;
184.72.64.0/19 1;
74.116.144.0/24 1;
216.115.21.0/24 1;
67.202.0.0/18 1;
23.131.144.0/24 1;
99.77.254.0/24 1;
54.160.0.0/14 1;
18.208.0.0/13 1;
208.79.47.0/24 1;
208.79.45.0/24 1;
52.4.0.0/14 1;
192.188.81.0/24 1;
52.70.0.0/15 1;
199.47.129.0/24 1;
54.86.0.0/16 1;
192.184.73.0/24 1;
67.226.221.0/24 1;
64.45.132.0/24 1;
107.23.0.0/17 1;
199.188.156.0/24 1;
142.54.39.0/24 1;
54.226.0.0/15 1;
52.54.0.0/15 1;
2600:1f18::/33 1;
168.149.246.0/24 1;
104.193.187.0/24 1;
74.221.131.0/24 1;
35.168.0.0/13 1;
142.54.38.0/24 1;
54.152.0.0/16 1;
64.238.7.0/24 1;
54.164.0.0/15 1;
208.84.160.0/24 1;

itoffshore commented 4 years ago

For blocking ASN's you should be using ipset

RealSuprim commented 4 years ago

How do i use it?

itoffshore commented 4 years ago

ipset man page

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Content scrapers not getting blocked. #335