search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.81k stars 472 forks source link

[BUG] Disco blocking Discordbot #441

Closed jadeops closed 2 years ago

jadeops commented 2 years ago

I use this list https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list to block user agents in the application layer instead of nginx.

After wasting several hours trying to figure out why Discord was unable to access open graph meta tags. I realised that Disco from the list was blocking Discordbot

Please check with this user agent "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)" if it affects nginx on your server.

mitchellkrogza commented 2 years ago

I do thorough testing for false positives and introduced word boundaries to prevent this as you can see Discord and Discordbot both are allowed through.

[~]$ curl -I https://mitchellkrog.com -A "Disco"
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

[~]$ curl -I https://mitchellkrog.com -A "Discord"
HTTP/2 200 
server: nginx
date: Sat, 31 Jul 2021 10:28:28 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-powered-by: PHP/7.4.14
last-modified: Sat, 03 Jul 2021 07:35:19 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0, report-uri='https://mitchellkrog.report-uri.com/r/d/ct/reportOnly'
referrer-policy: strict-origin-when-cross-origin
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block

[~]$ curl -I https://mitchellkrog.com -A "Discordbot"
HTTP/2 200 
server: nginx
date: Sat, 31 Jul 2021 10:29:26 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-powered-by: PHP/7.4.14
last-modified: Sat, 03 Jul 2021 07:35:19 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0, report-uri='https://mitchellkrog.report-uri.com/r/d/ct/reportOnly'
referrer-policy: strict-origin-when-cross-origin
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block

also tested with your full UA

[~]$ curl -I https://mitchellkrog.com -A "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)"
HTTP/2 200 
server: nginx
date: Sat, 31 Jul 2021 10:30:44 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
x-powered-by: PHP/7.4.14
last-modified: Sat, 03 Jul 2021 07:35:19 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0, report-uri='https://mitchellkrog.report-uri.com/r/d/ct/reportOnly'
referrer-policy: strict-origin-when-cross-origin
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
jadeops commented 2 years ago

Thanks! Word boundaries was what I missed. :+1: