search

amir-jakoby / crawler-commons

Automatically exported from code.google.com/p/crawler-commons
0 stars 0 forks source link

Suppport Google's "noindex" extension to robots.txt #19

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
According to this site, Google supports a "noindex: xxx" directive:

http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex
-your-site/

Original issue reported on code.google.com by kkrugler...@transpac.com on 17 Mar 2013 at 6:31