search

bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)
MIT License
6.69k stars 876 forks source link

Limit queue size #468

Closed ionutpopa closed 6 months ago

ionutpopa commented 6 months ago

Hello, I do not have an issue but I wanted to send this idea here: Can we have a limit for the queue size? For example I do not want to have too big of a queue size so I am putting a limit parameter into the queue function. Is there an alternative to this until we'll have something like that?

mike442144 commented 6 months ago

Better to handle outside crawler? because of two questions:

  1. what you hope to do when the queue is over limit?
  2. how/who to handle?in crawler itself or the developer?
ionutpopa commented 6 months ago

Well in my case to answer the first question is that when the queue limit is exceeded than I will not accept any more links to crawl but I did that outside the crawler as you said, thank you.