"Realm skip http://host.com/*" is ignored, when indexer received Crawl-delay
directive from host.com already from robots.txt
In Also ignored:
Robots no
Realm skip http://host.com/*
or
Robots no
Server site http://host.com/
indexer keep crawl-delay'ing...
Original issue reported on code.google.com by kogcha...@gmail.com on 25 Dec 2012 at 4:41
Original issue reported on code.google.com by
kogcha...@gmail.com
on 25 Dec 2012 at 4:41