search

nietaki / crawlie

A simple Elixir library for writing decently-performing crawlers with minimum effort.
MIT License
89 stars 11 forks source link

Fix the `:url_manager_timeout` logic #2

Closed nietaki closed 7 years ago

nietaki commented 7 years ago

Even for longer (~5s) values of url manager timeout, the UrlManager gracefully shuts down even though it still processes new urls.

Either fix the logic or find an approach that doesn't depend on the timeout.

nietaki commented 7 years ago

The right solution here is probably signal the finished and the failed fetches to the UrlManger.

nietaki commented 7 years ago

On a second thought keeping this as a bug and created https://github.com/nietaki/crawlie/issues/7 to track the change of approach.