basnijholt / adaptive-scheduler

Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky, or dask-mpi. :tada:
http://adaptive-scheduler.readthedocs.io
BSD 3-Clause "New" or "Revised" License
26 stars 10 forks source link

Calls to scheduler are not asynchronous #217

Open jbweston opened 8 months ago

jbweston commented 8 months ago

The Scheduler methods 'queue' and 'start_job' are blocking functions.

'start_new_jobs' is run in a ThreadPoolExecutor inside JobManager, but 'queue' is never run in a separate thread, and will block the event loop.

One fix would be to use 'asyncio.run_in_thread' everywhere these methods are called. An alternative is to make these methods async.

The first fix is easier to do, but also easier to footgun (forgetting to 'run_in_thread' impacts performance). Second fix would require more changes.

basnijholt commented 8 months ago

The reason I call start_new_jobs in a ThreadPoolExecutor is because when profiling, this part took a lot of time. queue seemed to be pretty fast.

Open to all improvements though!