Need to add `/robots.txt` to gateways

ipfs / kubo

An IPFS implementation in Go

Other

15.99k stars 3k forks source link

Open jbenet opened 8 years ago

jbenet commented 8 years ago

would be nice if gateways handled:

> curl 127.0.0.1:8080/robots.txt
User-agent: *
Allow: /

notes:

allow or disallow API?
this should be optional (config var?)
this should redirect robots to our main gateway https://ipfs.io to have a canonical location for all URLs found that will be properly rerouted with browser implementations/extensions.

jbenet commented 8 years ago

[ ] consider how this plays with header from #1747
[ ] perhaps make an option like Gateway.Robots.AllowIndexing, default to false.
[ ] option drives this header, and /robots.txt.
[ ] sharness tests for the config option, expected header, and /robots.txt