search

fredwu / crawler

A high performance web crawler / scraper in Elixir.
939 stars 91 forks source link

When finished #26

Closed umuro closed 4 years ago

umuro commented 4 years ago

How can we detect that the crawler exhausted crawling the site?

fredwu commented 4 years ago

Hi, there currently is no way of detecting it. However, in theory it shouldn't be too difficult to implement. Essentially a timer can be implemented so that if within x amount of time there's no more crawl then the job is considered done.

I'm no longer actively working on this project so PRs are welcome to address this. :)