Saving unreasonably slowly in headless

[sbudoff]

Hi,

I am a big fan of your stitching plugin and have integrated it into a python workflow for handling a somewhat complex image processing pipeline. In general the plugin works as expected with the virtual machine, however, saving the image at the end takes well over 2 hours where saving the same stack in regular FIJI is less than 10 minutes. I believe this is happening because each image plane is being generated sequentially, and that saving operation alone takes 1-2 minutes/plane. Am I implementing the plugin in python incorrectly? If not is there any way to speed up the saving such as parellelizing the plane saving?

Relevant code is as follows:

# Calculate available memory, leaving 10% aside
mem = psutil.virtual_memory().total / (1024 ** 3)  # Convert bytes to GB
mem = str(round(0.9*mem)) # Reduce memory allowance to 90 percent of system RAM
scyjava.config.add_options(f'-Xmx{mem}g') # Setting 90% of system RAM dynamically
ij = ij.init('sc.fiji:fiji:2.14.0') # Be sure FIJI is loaded as imageJ alone does not have the plugin
ij.getApp().getInfo(True)
for merged_dir in merged_dirs:
    # Stitch
    plugin = "Grid/Collection stitching"
    args = {
            "type": "[Positions from file]",
            "order": "[Defined by TileConfiguration]",
            "directory" : merged_dir,
            "layout_file" : "tile_locations.txt" ,
            "fusion_method": "[Linear Blending]",
            "regression_threshold": "0.30",
            "max/avg_displacement_threshold": "2.50",
            "absolute_displacement_threshold": "3.50",
            "compute_overlap": True,
            "subpixel_accuracy": True,
            "computation_parameters": "[Save computation time (but use more RAM)]",
            "image_output": "[Write to disk]",
    }
    ij.py.run_plugin(plugin, args)

Thank you!

[sbudoff]

To anyone that is trying to do something similar, here is the solution:

plugin = "Grid/Collection stitching"
for merged_dir in merged_dirs:
    subdir = extract_subdir(merged_dir)
    # Check if the tile_locations.registered.txt file already exists
    registered_file_path = os.path.join(merged_dir, "tile_locations.registered.txt")
    if os.path.exists(registered_file_path):
        print(f"Registration already performed, skipping registration step for directory: {merged_dir}")
        args = {
        "type": "[Positions from file]",
        "order": "[Defined by TileConfiguration]",
        "directory": merged_dir,
        "layout_file": "tile_locations.registered.txt",
        "fusion_method": "[Linear Blending]",
        "compute_overlap": True,
        "subpixel_accuracy": True,
        "computation_parameters": "[Save computation time (but use more RAM)]",
        "image_output": "[Keep output virtual]",
        }
    else:
        # Perform registration
        args = {
            "type": "[Positions from file]",
            "order": "[Defined by TileConfiguration]",
            "directory": merged_dir,
            "layout_file": "tile_locations.txt",
            "fusion_method": "[Linear Blending]",
            "regression_threshold": "0.30",
            "max/avg_displacement_threshold": "2.50",
            "absolute_displacement_threshold": "3.50",
            "compute_overlap": True,
            "subpixel_accuracy": True,
            "computation_parameters": "[Save computation time (but use more RAM)]",
            "image_output": "[Keep output virtual]",
        }
    ij.py.run_plugin(plugin, args) # Stitch the image
    print(f"Image {subdir} Stitch successful")
    output = ij.py.active_imageplus() # Extract image array from IJ
    print(f"Converting Image {subdir} from JAVA to Python")
    img = ij.py.from_java(output) # Convert JAVA array to python
    output.close() # Close the ImageJ image window to free up memory
    img = img.values # Convert byte array to 16 bit numpy array
    img = reorganize_array_for_imagej(img, current_axes='ZYXC', target_axes='ZCYX')
    print(f"Conversion of {subdir} from JAVA to Python SUccessful, now saving")
    metadata = meta_dict_4D[subdir]
    metadata['Width'] = img.shape[3]
    metadata['Height'] = img.shape[2]
    tif_format_saver(os.path.join(merged_dir, f"Stitched_4D_ZStack_{subdir}.tif"), img, metadata=metadata, axes='ZCYX')
    print(f"Image {subdir} 4D Stitch Saved Successfully")
    del img, output
    gc.collect()