search

alaz / legitbot

🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?
Other
24 stars 11 forks source link

Automate the process of updating IPs #57

Closed alaz closed 2 years ago

alaz commented 2 years ago

Some crawlers publish lists of IP addresses and they change frequently. Notably, DuckDuckGo: pull requests