Skip or Robots directive is ignored for hosts, having crawl-delay in robots.txt

shadisabzali / dataparksearch

Automatically exported from code.google.com/p/dataparksearch

GNU General Public License v2.0

0 stars 0 forks source link

Skip or Robots directive is ignored for hosts, having crawl-delay in robots.txt #45

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago

"Realm skip http://host.com/*" is ignored, when indexer received Crawl-delay 
directive from host.com already from robots.txt

In Also ignored:

Robots no
Realm skip http://host.com/*

or 

Robots no
Server site http://host.com/

indexer keep crawl-delay'ing...

Original issue reported on code.google.com by kogcha...@gmail.com on 25 Dec 2012 at 4:41