recrawling stops after all links in the frontier are crawled

VIDA-NYU / ache

ACHE is a web crawler for domain-specific search.

http://ache.readthedocs.io

Apache License 2.0

454 stars 135 forks source link

recrawling stops after all links in the frontier are crawled #159

Closed ashabbir closed 6 years ago

ashabbir commented 6 years ago

with recrawl config setup link_storage.recrawl_selector: MinRelevanceRecrawlSelector link_storage.recrawl_selector.relevance.min_relevance: 299 link_storage.recrawl_selector.relevance.interval: 1

The crawler stoping recrawling after all links in the frontier are crawled