search

joblib / loky

Robust and reusable Executor for joblib
http://loky.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
538 stars 45 forks source link

Prioritization or somewhat controlled ordering of submitted tasks? #291

Open josephschorr opened 3 years ago

josephschorr commented 3 years ago

I'd first like to thank the maintainers of the library: it has solved a number of deadlocks and other issues I encountered while attempting to run multiprocess code

I'm making use of loky as part of an engine which has jobs submitted to it at random times: these jobs are sharded to run as tasks in parallel via loky, thus making maximum use of available CPU resources whenever possible. However, currently, the order in which tasks submitted to loky runs appears to be fixed, which can result in a later job being "starved"/blocked until a sufficient set of submitted tasks for the former job(s) has completed. While I can manually mitigate this somewhat by reducing the use of available processes for a specific job, this leaves CPU cores unused. Is there a known way to indicate to loky that certain submitted tasks have a higher priority, or to perhaps submit tasks to be placed at the front of the queue, rather than the back? My brief delve into the Queue implementation within loky suggests the answer is "no" (as then it would be a stack, rather than a queue), but I wanted to confirm.

tomMoral commented 3 years ago

Hello and thanks for the kind words for loky :)

For now, loky only provides a fix ordering for the task indeed. This part is mainly due to the use of a queue.Queue when we submit a new task.

However, I don't think this is very complex to change this behavior. Indeed, when a job is submitted, it is put in a local _work_ids queue with put. Then when there is some free space for computations, a task is fetch from this queue (get(block=False), throws queue.Empty error when empty) and send to the pool of workers (through a queue, this one cannot be changed but I don't think this matter for what you describe). If you change the work_ids queue with another structure that implements these two methods (put/get(block=False)) I don't see any major issuers. This would change the ordering with which the task are run. As this is local, there should be no concurrence issues. If you implement such strategy, I would be happy to see the result.

josephschorr commented 3 years ago

@tomMoral I'll definitely take a stab at implementing that then. Will send a PR if I get it working :)

josephschorr commented 3 years ago

@tomMoral https://github.com/joblib/loky/pull/292. Let me know if I made any bad assumptions or didn't follow a style somewhere

josephschorr commented 3 years ago

@tomMoral Let me know if you need me to make any changes on #292 in order to get it in