Closed beeirl closed 5 years ago
Hi @Druux, yes, the local RequestQueueLocal
will grow to infinity. The original purpose of it was to enable local development and testing, rather than to provide a full-featured production level storage. But people seem to be using it this way, so I'd be happy to review a PR that adds this functionality.
@mnmkng Thanks for your feedback. Sure, I will submit a pull request as soon as I got it working.
Good luck then!
Hello, everybody. I am using a
RequestQueue
using theLocalStorage
for myCheerioCrawler
. I wonder if the number of generated .json files in thehandled
folder can increase to infinity or be deleted automatically at some point. I'm asking because I have to handle a dynamic set of URLs in one crawl, that can be very large under certain circumstances.