search

elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
965 stars 113 forks source link

Suggestion for nameing of closespider_timeout in config #183

Closed EdmondFrank closed 1 year ago

EdmondFrank commented 3 years ago

image

It seems that when all the workers’ requests list are empty, the crawler still cannot stop automatically.

Maybe it is more convenient to determine when the spider need be stopped by providing a optional item closespider_deltacount with default value 0 in config

EdmondFrank commented 3 years ago

This issue can be solved by setting closespider_timeout: 0

But I feel that the name of this settings item is confusing.

It is easy to misunderstand that this is a setting about the timeout of network requests

Maybe closespider_min_delta is better, this is just my personal suggestion.

Ziinc commented 3 years ago

I see, yes, setting naming is an issue, but that's what the docs are for :sweat_smile: Interesting that the spider doesn't auto close when there isn't any requests in the queue, might be an issue with default config.