search

vanvalenlab / deepcell-applications

Run DeepCell Applications
Other
8 stars 6 forks source link

Batch processing #23

Closed jil24 closed 2 years ago

jil24 commented 3 years ago

It's unclear how this currently implements batch processing. I see the batch-size parameter, but passing in a multi-slice tiff (single channel, for nuclear segmentation) only processes the first slice. I currently have a pipeline where the entire process relaunches between slices; I assume this is inefficient.

ngreenwald commented 3 years ago

That's correct, we currently don't have batch processing of multiple images implemented here. The user feedback we got for an executable docker was that people were interested in plugging this into a larger image analysis pipeline. Generally those pipeline managers process each image separately, to allow for better parallelization across compute resources.

Do you have another use case in mind where it would instead be beneficial for the CLI to process multiple images at once?

jil24 commented 3 years ago

Not at all. If the other approach isn’t wasteful of resources it’s quite easy to do.

ngreenwald commented 3 years ago

There are definitely some efficiency gains from processing multiple images in parallel, but the whole thing runs pretty fast. If it ends up being prohibitively slow, let me know.