BRT task: gen_ng_metadata takes long time

niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"

BSD 3-Clause "New" or "Revised" License

5 stars 1 forks source link

BRT task: gen_ng_metadata takes long time #411

Closed annshress closed 8 months ago

annshress commented 8 months ago

Flow run : https://prefect2.hedwig-workflow-api.niaiddev.net/flow-runs/flow-run/983440ff-7736-458f-b0c9-29da2fa7aa11

annshress commented 8 months ago

The issue is Zarr Image.neuroglancer_shader_parameters() is taking longer than expected and is being killed by time limit of job queueing system set by us (~ 4hours).

~~This is the case when there are a number of "Building histogram for zarr groups" being run at the same time.~~

This happens when there are large number of samples (30+), even though the function is being invoked once at a time till completion.

This normally only takes a couple of seconds.

annshress commented 8 months ago

Solution for now: Increased the number of cores used by dask task runner (from 20 to 64).

Needs more investigation on why multiple tasks sequentially sharing the core is problematic!