search

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders
Other
777 stars 172 forks source link

diffbot #54

Closed ghost closed 6 years ago

ghost commented 6 years ago

Hi, I saw some diffbot activity in my access.log

35.188.153.21 - - [time] "GET / HTTP/1.1" 200 33374 "http://news.google.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)"

Its could be bad bot. Its have been listed some networks.

https://www.distilnetworks.com/bot-directory/bot/diffbot/ http://soggi.eu/misc/articles/Diffbot-blocking-bad-bot-rude-content-scraper-from-websites.htm

also, its been our bad referer list. (diffbot.com)

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-referrers.list

mitchellkrogza commented 6 years ago

Hi @yakusha sorry for the delay, been sick for the past 10 days but on the mend. Diffbot from my research is an data collecting company who uses this information gathered from web sites to help customers of theirs who pay for their services. Check their web site at https://www.diffbot.com/ generally if I find things like this in my logs and I see it is a paid for service I block them as do I block most SEO companies out there too.

mitchellkrogza commented 6 years ago

@yakusha any feedback on this or can I close this issue?

ghost commented 6 years ago

you can close ticket. sorry.

On Jan 5, 2018 14:25, "Mitchell Krog" notifications@github.com wrote:

@yakusha https://github.com/yakusha any feedback on this or can I close this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/issues/54#issuecomment-355554030, or mute the thread https://github.com/notifications/unsubscribe-auth/AF7jt0SEGrAFlyaQAO_61nduCNvNbaKKks5tHiLigaJpZM4QW_mG .

mitchellkrogza commented 6 years ago

No worries, closing. All the best for 2018