sunu
commented
3 years ago sync mode is now the default mode of execution and supports multiple threads. If a large crawler is still run in single threaded mode, please make sure to use tail call recursion in the pipeline to avoid queue OOM issues.
In sync mode, memorious execute tasks linearly in a single thread. Crawlers that loop through large number of entries, put too many tasks on the queue before they can move on to the next stage and pull tasks out of the queue.
This often results in larger crawler running out of memory when running in sync mode.
One solution to this problem is to execute tasks in a depth-first fashion as they come without putting them on the task queue when executing in sync mode.