search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.82k stars 472 forks source link

[BUG] Whitelisting User Agents don't work and still get blocked with 444 #372

Closed disaster123 closed 4 years ago

disaster123 commented 4 years ago

Describe the bug

A customer is using the service from Host-Tracker.com but the service is blocked with 444 in the nginx Logfile.

To Reproduce

Setup an Account at Host-Tracker.com and let it visit your website.

Expected behavior

I should be able to whitelist HostTracker with: "~*(?:\b)HostTracker(?:\b)" 0;

inside /etc/nginx/bots.d/blacklist-user-agents.conf

Paste any log / error messages here (paste in between the markers) just replaced some hex code with XXXX

37.221.113.21 - - [15/Apr/2020:20:05:13 +0200] "GET / HTTP/1.1" 444 0 "http://www.host-tracker.com/UptimeGraph/Graph/2ae3e3c2-d111-4a34-XXXX-9362e70bd2d9" "Mozilla/5.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)"

@mitchellkrogza any way to whitelist these?

mitchellkrogza commented 4 years ago

Morning @disaster123 whitelisting does work and all over-rides are fully tested during every build - here you can see Sistrix being whitelisted perfectly - https://travis-ci.org/github/mitchellkrogza/nginx-ultimate-bad-bot-blocker/jobs/675311633#L7195-L7196 and host-tracker.com https://travis-ci.org/github/mitchellkrogza/nginx-ultimate-bad-bot-blocker/jobs/675311633#L7730-L7731

Travis CI - Test and Deploy Your Code with Confidence
Travis CI enables your team to test and ship your apps with confidence. Easily sync your projects with Travis CI and you'll be testing your code in minutes.
Travis CI - Test and Deploy Your Code with Confidence
Travis CI enables your team to test and ship your apps with confidence. Easily sync your projects with Travis CI and you'll be testing your code in minutes.
disaster123 commented 4 years ago

Ah OK Sistrix did not work due to nginx map cache i only did a reload instead of restart.

Regarding Host-Tracker i tried to whitelist the User Agent - but it seems i need to whitelist the domain or referrer?

mitchellkrogza commented 4 years ago

Always need to reload nginx sudo nginx -t && sudo nginx -s reload

With host-tracker you will need to whitelist the User-Agent and Referrer/Domain