When we are distributing jobs to different threads, the current approach is to use Iterators.partition to split the workload to all available threads, but Iterators.partition returns a Vector, and it shows it is actually pretty slow in doing so for large amount of items.
But by visualizing the problem in the perspective of consumer-supplier, we can see that we don't have to split the work evenly at start, and just let all threads grab their job concurrently until there's nothing left, and for this we can use Channel, which is thread-safe.
This issue does not guarantee a fix since we have to evaluate the performance against the existing module.
When we are distributing jobs to different threads, the current approach is to use
Iterators.partition
to split the workload to all available threads, butIterators.partition
returns aVector
, and it shows it is actually pretty slow in doing so for large amount of items. But by visualizing the problem in the perspective of consumer-supplier, we can see that we don't have to split the work evenly at start, and just let all threads grab their job concurrently until there's nothing left, and for this we can useChannel
, which is thread-safe.This issue does not guarantee a fix since we have to evaluate the performance against the existing module.