search

bejean / crawl-anywhere

Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.
www.crawl-anywhere.com
Apache License 2.0
96 stars 38 forks source link

88: Add per-host config for bypassing robots file #91

Open grimsa opened 7 years ago

grimsa commented 7 years ago

Resolves issue https://github.com/bejean/crawl-anywhere/issues/88