search

benjojo / gophervista

It's like AltaVista, but for RFC 1436 Gopher sites
http://gophervista.benjojo.co.uk
MIT License
55 stars 2 forks source link

robots.txt #4

Open jamestomasino opened 6 years ago

jamestomasino commented 6 years ago

Just a note that if you are publishing your crawler results to please respect sites which host a robots.txt restricting spidering.

For instance: gopher://gopher.black/0/robots.txt

Thanks

benjojo commented 6 years ago

Is robots an accepted standard for Gopher? I feel like there are quite a few edge cases around taking something for HTTP for Gopher

jamestomasino commented 6 years ago

It's being adopted more and more as search engines spider gopherspace via proxies. There's no standard for robots since there's no standard for web-based search engines in gopherspace. Robots seem to be the most rational response, though.