search

czbiohub-sf / shrimPy

shrimPy: Smart High-throughput Robust Imaging & Measurement in Python
BSD 3-Clause "New" or "Revised" License
7 stars 1 forks source link

multiprocessing model for mantis #27

Open mattersoflight opened 1 year ago

mattersoflight commented 1 year ago

We need a few acquisition (label-free and light-sheet) and analysis (viewer) processes to run in parallel. In the future, it may be useful to have other features (FOV scoring, auto-alignment) run in their separate threads.

We would want a multiprocessing library that provides an intuitive Pythonic implementation of synchronization primitives (semaphores and mutex) to facilitate concurrent execution when we need it.

@ieivanov What is the current multiprocessing model for mantis and should the model be revised? You mentioned that pycro-manager spins up a process for every hook function - what library does pycro-manager use?

Qt has support for semaphores and mutex.

@ziw-liu can you point to an example or two of how recOrder uses qt threads for synchronization?

ziw-liu commented 1 year ago

@ziw-liu can you point to an example or two of how recOrder uses qt threads for synchronization?

When recOrder performs calibration, the real-time progress from the calibration worker thread is emitted to a Qt signal slot in the main thread (napari event loop):

https://github.com/mehta-lab/recOrder/blob/802cc62cac537749ab5dce5535d7c0a18137f819/recOrder/plugin/main_widget.py#L1852-L1853

ieivanov commented 1 year ago

I actually don't think there is a large outstanding need for multiprocessing in the mantis acquisition engine.

Multiprocessing is most useful when the individual processes perform independent tasks in parallel. Inter-process communication is possible, but I think not very straightforward.

The LS and LF acquisitions already run in parallel on separate processes by virtue of running on two separate MM instances.

FOV scoring and autofocus would need to run on the same thread as the acquisition engine - the acquisition would need to wait until these are finished, so I think there is no need to spin up another process. If autofocus needs to perform heaving math before returning, we could consider putting it on another process while the acquisition moves on to the next position, but that would complicate the acquisition flow quite a bit.

A napari viewer is a good example of where multiprocessing is helpful. The acq engine can keep on acquiring data while it is being viewed in another process. I am planning to implement that using the subprocess module. That is still to be worked out and tested.