Task Priority

[Causeless]

Hey, so I'm having an issue with BS::thread_pool for the purposes of a game that does many async requests. These requests can cause huge framespiking because tasks are always taken on by worker threads in order of submission, and tasks are never swapped away from. For example:

1. Pathfinding is performed in an asynchronous manner in a background thread. We can happily wait multiple frames or even several seconds for a request to complete.
2. We also have Lua GC performed in a background thread, so that the GC can run during the background in otherwise non-threaded parts of the code instead of slowing down the main Lua updates. However, it's crucial that any requested GC operation must be complete by the next frame before our next Lua update.

Right now, the worker threads get saturated with pathfinding tasks and then waiting on the Lua GC request at the back of the queue enforces the entire game to stop and wait for all pathfinding requests to finish, causing massive framespikes.

Is it possible to differentiate task priority and/or thread-pool priority? Generally any task .wait()-ed upon I'd hope to be performed ASAP However, for my specific usage case, not only would we need to be able to push these Lua GC tasks to the front of the queue, but we need to actively preempt thread priority away from the pathfinding tasks to handle the higher-importance tasks.

Given that tasks cannot be terminated early or swapped from, I imagine I'd need to use two thread pools. In that case, would it be possible to give one threadpool priority to be preferentially scheduled towards? I'm not sure if this is reasonably doable and certainly unsure about if it's reasonably doable in a portable manner, but it would be very useful.

Furthermore, it might be useful to just ignore the Lua GC this frame and just continue, until the thread gets around to dealing with it. For this, I'd need to be able to pause (or purge) only a specific task to stop it from starting during the Lua update, without pausing the entire threadpool or affecting other tasks. Is this possible?

[Causeless]

Another issue I'm having is enormous pauses that seem to happen for no reason. For example, in the uploaded image you can see "LuaMan::Update() taking obscene amounts of time, on the order of 5ms, doing literally nothing expect for -wait() ing on tasks that have already finished (the Lua Garbage Collection tasks to the left, which are completed long before the function is even called). No other thread is busy and in fact CPU usage just is effectively nothing at this point.

EDIT:

Okay, fixed this one. The issue is that a multi_future doesn't clear the existing futures after waiting on them. I needed to reconstruct the multi_future after each use to stop unused pointless futures hanging around. On that note... any chance for a multi_future.clear()?

[Causeless]

Another feature request would be the ability to ask if a multi-future is complete without explicitly waiting on it, as well as general wait_until() and wait_for() on multifuture.

[jakeanq]

I'm working on a user interface which uses this thread pool for loading files from disk; similar functionality would be quite useful as I'd like to prioritise a certain type of file request (but continue loading the other files in the background).

Perhaps there could be a variant of the thread pool which uses a priority queue for tasks and allows an integer priority value to be passed when submitting tasks?

[bshoshany]

Hi @Causeless and @jakeanq and thanks for the suggestions! In fact, most of your suggestions are already being implemented in the upcoming v4.0.0 release.

Is it possible to differentiate task priority and/or thread-pool priority? Generally any task .wait()-ed upon I'd hope to be performed ASAP However, for my specific usage case, not only would we need to be able to push these Lua GC tasks to the front of the queue, but we need to actively preempt thread priority away from the pathfinding tasks to handle the higher-importance tasks. Given that tasks cannot be terminated early or swapped from, I imagine I'd need to use two thread pools. In that case, would it be possible to give one threadpool priority to be preferentially scheduled towards? I'm not sure if this is reasonably doable and certainly unsure about if it's reasonably doable in a portable manner, but it would be very useful.

1. The new release will allow access to the std::thread::native_handle() of each thread in order to implement OS-specific priority. This means you can have two thread pools, one with high priority and one with low priority. Note that this is not portable, but can be made portable if you detect the OS and use either pthread.h on Linux/macOS or windows.h on Windows.
2. The new release will implement a priority queue for the thread pool. This is something I'm currently working on and will most likely use a combination of std::priority_queue along with a normal std::queue for maximum performance. This means you can ensure that tasks with higher priority are executed from the queue first (note that this differs from OS priority, which is more about resource access than order of execution).

Furthermore, it might be useful to just ignore the Lua GC this frame and just continue, until the thread gets around to dealing with it. For this, I'd need to be able to pause (or purge) only a specific task to stop it from starting during the Lua update, without pausing the entire threadpool or affecting other tasks. Is this possible?

Terminating, pausing, or purging individual tasks is not something that can be implemented in the thread pool, since once the task is executed, the thread pool cannot intervene in its code. However, you can embed a "kill switch" (or "pause switch") in the task on your own, and check it occasionally within the internal loop(s) of the task itself.

The issue is that a multi_future doesn't clear the existing futures after waiting on them. I needed to reconstruct the multi_future after each use to stop unused pointless futures hanging around. On that note... any chance for a multi_future.clear()?

Sure, I can add that in v4.0.0.

Another feature request would be the ability to ask if a multi-future is complete without explicitly waiting on it, as well as general wait_until() and wait_for() on multifuture.

As far as I know, it is currently impossible in C++ to check if a future is complete without explicitly waiting on it. You can use wait_until() and wait_for() to check for that indirectly. I'll think about how this can be implemented for all futures in a multi_future.