search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.81k stars 472 forks source link

[BUG] Duplication in globalblacklist.conf #481

Closed cuiyang000 closed 5 months ago

cuiyang000 commented 2 years ago

Error:

`nginx: [warn] duplicate network "138.199.57.151", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18699

nginx: [warn] duplicate network "143.244.38.129", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18715

nginx: [warn] duplicate network "195.181.163.194", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18810

nginx: [warn] duplicate network "5.188.120.15", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18937

nginx: [warn] duplicate network "89.187.173.66", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18984`

mitchellkrogza commented 2 years ago

It's only a warning of a duplicate IP you can ignore it.

ludovicsclain commented 2 years ago

Hey there, I also have this warnings on NBBB updates:

nginx: [warn] duplicate network "138.199.57.151", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18699 nginx: [warn] duplicate network "143.244.38.129", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18715 nginx: [warn] duplicate network "195.181.163.194", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18810 nginx: [warn] duplicate network "5.188.120.15", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18937 nginx: [warn] duplicate network "89.187.173.66", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18984

I realize it's just duplicates and I can ignore it, but I'm working on optimizing my nginx server and having it on every reboot is quite troubling. Should I do something at my level or are you planning to remove these duplicates? Thank you for your help! Ludovic

mitchellkrogza commented 2 years ago

Can't remove the dupes as we have Bunny.cdn ranges being whitelisted and some are blacklisted by some services. See: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/476

ludovicsclain commented 2 years ago

Can't remove the dupes as we have Bunny.cdn ranges being whitelisted and some are blacklisted by some services. See: #476

Thank you for the very useful explanation, I find it strange that specific configurations are globalized to all, generating for example these alerts, personally I do not use bunny.net CDN 🤔 

lunaraurora commented 1 year ago

my duplicates are all inside this section of the file globalblacklist.conf:

START SEO ANALYSIS TOOLS ### DO NOT EDIT THIS LINE AT ALL

the list includes 5 IPs, all from bunnycdn; example: 138.199.57.151 - 143.244.38.129 - 195.181.163.194 - 5.188.120.15 - 89.187.173.66 with value 1 (meaning are blocked). Further down, i have inserted all the IPs of bunnycdn that i manually whitelisted (value 0) because i had problems that are fixed with this list. Inside this list there are all the 5 ip from bunny with value 0 that make conflict with the duplicates that have value 1, and producing such warnings. So my question is: could you set the SEO ANALYSIS TOOLS in way to ignore such list: https://api.bunny.net/system/edgeserverlist/plain ? Many Thanks

glcode80 commented 1 year ago

I ended up running a script via a cron job every time after the cron that updates globalblacklist.conf:

sed -i 's/^\t*138.199.57.151\t*1;$/#\t138.199.57.151\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*143.244.38.129\t*1;$/#\t143.244.38.129\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*195.181.163.194\t*1;$/#\t195.181.163.194\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*5.188.120.15\t*1;$/#\t5.188.120.15\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*89.187.173.66\t*1;$/#\t89.187.173.66\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf nginx -t service nginx reload

So simply commenting out the first appearance of the duplicates and then reloading nginx.

Quick and dirty. A more advanced solution would be a script to properly check for duplicates. As the 5 IPs here seem to be always the same it is working so far for me.

Of course, a much better solution would be, to run a simple check for duplicates on globalblacklist.conf before publishing the list. Are there any plans to do that?

lunaraurora commented 1 year ago

I ended up running a script via a cron job every time after the cron that updates globalblacklist.conf:

sed -i 's/^\t*138.199.57.151\t*1;$/#\t138.199.57.151\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*143.244.38.129\t*1;$/#\t143.244.38.129\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*195.181.163.194\t*1;$/#\t195.181.163.194\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*5.188.120.15\t*1;$/#\t5.188.120.15\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*89.187.173.66\t*1;$/#\t89.187.173.66\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf nginx -t service nginx reload

Working.. thank you!

alexgurrola commented 1 year ago

@glcode80 Thanks for the band-aid. Hopefully this will be fixed in the main soon.

tradenet commented 5 months ago

I wonder if this will be the year of the "Bunny" to address this issue. j/k

mitchellkrogza commented 5 months ago

This is NOT a bug, this is desired and perfectly normal behaviour and it cannot be fixed either. IP's are blacklisted from daily blacklists old value "1" which then gets whitelisted at the end of the blocker in the global whitelist value "0" these include CDN's and other important resources that need to be whitelisted.

These are simple [WARN] messages not [EMERG] and should just be ignored. They do NOT affect Nginx in any way.

mitchellkrogza commented 5 months ago

I ended up running a script via a cron job every time after the cron that updates globalblacklist.conf:

sed -i 's/^\t*138.199.57.151\t*1;$/#\t138.199.57.151\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*143.244.38.129\t*1;$/#\t143.244.38.129\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*195.181.163.194\t*1;$/#\t195.181.163.194\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*5.188.120.15\t*1;$/#\t5.188.120.15\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf sed -i 's/^\t*89.187.173.66\t*1;$/#\t89.187.173.66\t\t1;/g' /etc/nginx/conf.d/globalblacklist.conf nginx -t service nginx reload

So simply commenting out the first appearance of the duplicates and then reloading nginx.

Quick and dirty. A more advanced solution would be a script to properly check for duplicates. As the 5 IPs here seem to be always the same it is working so far for me.

Of course, a much better solution would be, to run a simple check for duplicates on globalblacklist.conf before publishing the list. Are there any plans to do that?

You need to understand the blocker. Removing these with sed is not a solution nor is it required. These are only [WARN] messages, they do not affect Nginx in any way whatsoever and this is perfectly normal behaviour.

Blacklists load BEFORE the Whitelists.

blacklist - value "1" - whitelist = value "0"

Read the message carefully value "0" = current value (whitelisted) old value "1" first loaded value which was (blacklisted).

Some IP's are blacklisted with daily blacklist updates, we then have some of those IP's whitelisted as they are important resources like CDN's etc.

glcode80 commented 5 months ago

Thanks @mitchellkrogza for the comments. I understand your point of view that this is only a warning and does not affect Nginx behavior. I can only speak for myself, but I always try to clean up my configs until I get rid of all the [WARN] messages too. This can be depreciation notices for example. I consider this best practice. If I check logs, I search for WARN messages too. As there is no simple way to get them removed for the bad-bot-blocker they show up again and again and they always get my attention. The only way is the hacky sed approach. I would really appreciate, if the duplicates could be removed already in globalblacklist.conf so that there are no more WARN messages. If not a priority for you, then I can live with that too. Much more important for me that you keep up the good work on this project, really appreciate a lot what you are doing here.

tradenet commented 5 months ago

Thanks @mitchellkrogza for the comments. I understand your point of view that this is only a warning and does not affect Nginx behavior. I can only speak for myself, but I always try to clean up my configs until I get rid of all the [WARN] messages too. This can be depreciation notices for example. I consider this best practice. If I check logs, I search for WARN messages too. As there is no simple way to get them removed for the bad-bot-blocker they show up again and again and they always get my attention. The only way is the hacky sed approach. I would really appreciate, if the duplicates could be removed already in globalblacklist.conf so that there are no more WARN messages. If not a priority for you, then I can live with that too. Much more important for me that you keep up the good work on this project, really appreciate a lot what you are doing here.

I agree. Us system admin types tend to have real OCD over these "sort of issues". Especially us "old school" types. Don't get us wrong, it's all good work. And we appreciate it. Thanks.