hoogenboom-group / interactive-render-workflow

Interactive workflow for aligning volume electron microscopy datasets.
GNU General Public License v3.0
0 stars 0 forks source link

`renderapi.client.pointMatchClient` breaks with `concurrent.futures.ThreadPoolExecutor` #16

Closed arentkievits closed 3 months ago

arentkievits commented 4 months ago

We use the futures module to run renderapi.client.pointMatchClient on multiple threads during stitching. This is because the tile pairs are cropped to only look for matches in the overlap regions. However, with this setting, only one thread is allowed inside renderapi.client.pointMatchClient, which would normally support multithreading.

However, the output of futures is not the same as running renderapi.client.pointMatchClient in a regular loop without multithreading:

With multithreading (futures) image

Without multithreading image

arentkievits commented 3 months ago
try:
    for tile_pair, pos in tqdm(zip(tile_pairs_reformat, relativePositions),
                               desc="submitting tile pairs",
                               total=len(tile_pairs_reformat),
                               unit="tilepairs",
                               smoothing=0.3):
        params_SIFT.firstCanvasPosition = pos
        future = executor.submit(
            renderapi.client.pointMatchClient,
            stack=stack,
            collection=match_collection,
            tile_pairs=[tile_pair],
            sift_options=params_SIFT,
            excludeAllTransforms=True,
            subprocess_mode='check_output',  # suppresses output
            **params_align)
        futures.add(future)

overwrites params_SIFT.firstCanvasPosition N times where N is the number of workers, because the memory is shared. So only the last value gets used. Because the tile pairs are alternatively on the rows and columns, the clipping is only correct for 50% of the tiles. This causes the error.

This would work in a multiprocess setup though

arentkievits commented 3 months ago

https://github.com/hoogenboom-group/interactive-render-workflow/commit/84b14aa573f18019aa1d82db84f03ebb0d0cb319 fixes this