search

tasfe / crawler-commons

Automatically exported from code.google.com/p/crawler-commons
0 stars 1 forks source link

Suppport Google's "noindex" extension to robots.txt #19

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
According to this site, Google supports a "noindex: xxx" directive:

http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex
-your-site/

Original issue reported on code.google.com by kkrugler...@transpac.com on 17 Mar 2013 at 6:31