bigsky as-a-crawler polish

bluesky-social / indigo

Go source code for Bluesky's atproto services.

Apache License 2.0

600 stars 88 forks source link

A couple things we should look at for bigsky, in the context of it reaching out and crawling an increasingly large number of hosts on the web.

[x] going to make HTTP(S) connections to random web servers. it can self rate-limit, but should ensure it has robust timeouts to not get honey-trapped
[ ] we should ensure that the "ding/dong" requests can't be used to make us DDoS other servers (outgoing requests)
[x] needs to have fresh SSL cert database when deployed: need to include in alpine docker image? or maybe just switch to ubuntu base for deployed images?
[ ] when bigsky connects to servers, it should set a User-Agent identifying itself, and linking back to some contact info (webpage about crawling, or email address). this is standard best practice for bots/crawlers
[ ] should think about robots.txt a bit. I'm not very concerned because we mostly crawl on request, and are connecting at an atproto-specific URL (/xrpc/...)

Crawler user-agent with some content info included is the next highest priority, IMO.

Here are some example User-Agents, showing how contact info is included:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
webprosbot/2.0 (+mailto:abuse-6337@webpros.com)
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
Dreamwidth Studios (webmaster@dreamwidth.org; for http://www.dreamwidth.org/users/bnewbold_links_feed/; 1 readers)
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)

bluesky-social / indigo