I'm thinking of extending the parallelDedupQueue to work as a normal queue. Basically, each actor would have one of the queues for reading and would be adding requests to other queues depending on the hash of the uniqueKey(like the dedup one does).
Just need to figure out the details of how the handleTasks would be spawned:
The crawler must not finish on an empty queue. It can only finish when all queues are done. We can have an interval that checks that once a minute or so and if yes, it shuts down the crawler.
We don't want the crawler that has an empty queue to stay in a busy loop checking for the next request. Need to have some delay there. Need to check the exact implementation in the SDK.
I'm thinking of extending the parallelDedupQueue to work as a normal queue. Basically, each actor would have one of the queues for reading and would be adding requests to other queues depending on the hash of the uniqueKey(like the dedup one does).
Just need to figure out the details of how the handleTasks would be spawned: