Open jbenet opened 8 years ago
would be nice if gateways handled:
> curl 127.0.0.1:8080/robots.txt User-agent: * Allow: /
notes:
https://ipfs.io
from https://github.com/ipfs/go-ipfs/pull/1747/files#r40535116:
Gateway.Robots.AllowIndexing
/robots.txt
would be nice if gateways handled:
notes:
https://ipfs.io
to have a canonical location for all URLs found that will be properly rerouted with browser implementations/extensions.