apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.23k stars 3.47k forks source link

[C++] Create work stealing implementation of generalized ThreadPool #28631

Open asfimport opened 3 years ago

asfimport commented 3 years ago

Given the chase-lev deque and the generalized thread pool we are now able to create a work stealing version.  This issue forces us to address some thorny issues that don't concern academic thread pools.

For example, we need to allow tasks for the thread pool to be submitted from outside the thread pool itself.  An academic work-stealing thread queue has one queue per thread and each thread adds incoming tasks to its own queue.  If the task adding queue isn't in the thread pool then it doesn't have a queue.  This is a tricky issue because it implies a queue with multiple producers and multiple consumers.

There's a couple of ways to tackle this but generally it means that the work stealing thread pool is going to have to rely on locking in a number of places.  Ideally the hot path can be kept lock free.

Reporter: Weston Pace / @westonpace

Note: This issue was originally created as ARROW-12902. Please see the migration documentation for further details.

asfimport commented 2 years ago

Todd Farmer / @toddfarmer: This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.