niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"
BSD 3-Clause "New" or "Revised" License
5 stars 1 forks source link

Neuroglancer metadata generation task deadlocks in BRT flow #478

Closed blowekamp closed 1 month ago

blowekamp commented 1 month ago

When the number of BRT images exceeds the number of workers all workers are busy for extended periods of time. The pytools use dask.array which by default use the Prefect's Dask Distributed scheduler. This causes a nested usage of the scheduler and the pytools dask.array task to lock a Dask workers into a deadlock state with not free workers to do the requested task.

The proposed solution is for pytools dask used to use a local threaded scheduler to avoid shared usage with the Prefect scheduler.