search

alaz / legitbot

🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?
Other
24 stars 11 forks source link

Custom Rubocop cop to update crawler IPs [issue #57] #64

Closed alaz closed 2 years ago

alaz commented 2 years ago

ip_ranges may define URL and a CSS selector of the page with up-to-date list of IP ranges, e.g.

https://github.com/alaz/legitbot/blob/fdbae704ec3fffdc3483087b1fe251104ade519b/lib/legitbot/duckduckgo.rb#L6-L8