Open fpietrosanti opened 12 years ago
Tor2web represent the access to a Darknet.
Into a Darknet there's no Google Search Engine.
We should implement a default robots.txt to prevent google from indexing both webpages and images on Tor2web sites.
https://www.google.com/support/webmasters/bin/answer.py?answer=156449
The option to provide a static Robots.txt preventing the site from working must be configurable.
Tor2web represent the access to a Darknet.
Into a Darknet there's no Google Search Engine.
We should implement a default robots.txt to prevent google from indexing both webpages and images on Tor2web sites.
https://www.google.com/support/webmasters/bin/answer.py?answer=156449
The option to provide a static Robots.txt preventing the site from working must be configurable.