alaz / legitbot

🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?
Other
24 stars 11 forks source link

Crawlers are banned by GPTBot #131

Closed alaz closed 5 months ago

alaz commented 5 months ago

lint action is failing:

lib/legitbot/gptbot.rb:1:1: C: Custom/IpRanges: Could not fetch IPs from https://openai.com/gptbot-ranges.txt , HTTP status code 403

It works when I run it locally, hence I suspect they have banned Microsoft or Github Actions IP ranges.

I let them know on X (formerly Twitter), but frankly I do not expect much: https://twitter.com/aazarov/status/1790281032513507420