Registration V2

[edyoshikun]

This PR adds the following:

• CLI for manual estimation of affine transform
• CLI for optimizing an affine transformation with ants
• CLI for applying a transformation using ants and cropping to overlapping region using Largest Internal Rectangle (LIR)

TODO:

• [x] merge #118 (need to push updates from fry2)
• [x] test that registration works with keep_overhang: true @edyoshikun
• [x] now that estimate_affine, optimize_affine, and apply_affine all take source and targets dirpaths, we no longer need to keep source_shape_zyx and target_shape_zyx in the yml config file @edyoshikun
• [x] refactor registration and stabilization functions out of utils and into the registration and stabilization modules @ieivanov
• [x] clean up function names @ieivanov
• [x] update slurmkit examples
• [x] link issues closed by this PR

Future PRs:

• [ ] rename target to fixed and source to moving to match convention in other image registration pipelines
• [ ] rename estimate-affine, optimize-affine, and apply-affine CLI calls to estimate-registration, optimize-registration, and register
• [ ] refactor process_single_position_v2 and apply_transform_to_zyx_and_save_v2 (and affine_transform) to work on arrays of consistent shape - either ZYX or CZYX, not both. Possibly move process_single_position and apply_transform_to_zyx_and_save to iohub (long with update_scale_metadata?). @talonchandler @ziw-liu
• [ ] try to find better cropping algorithm (maybe https://github.com/planetlabs/maxrect or https://shapely.readthedocs.io/en/stable/reference/shapely.intersection.html)
• [ ] check how apply_affine handles positions will all nans or zeros. Currently we convert nans to zeros, which should not be necessary

[ieivanov]

In the final version of the registration pipeline we should rotate the volumes by 90 deg such that they are in landscape mode rather than portrait mode (i.e. with larger width than height). Our brains process landscape aspect ratio more naturally and our monitors are built that way too. This is not critical now, but may affect some of the earlier design choices.

[edyoshikun]

@talonchandler let's schedule some time to go over this. Some design choices that we should also think about:

• Use scipy.ndimage vs ants registration
• What yaml config files should we take as input and output for the estimate-affine and optimize-affine.
• I don't want to complicate the functions to add the 90 deg rotation. If it's only for display, should we add this as a coordinateTransformMetadta()?

[talonchandler]

Excellent! @edyoshikun and I just finished a paired session and put together this list of (very superficial) TODO items. Thanks for all of your effort here @edyoshikun.

@edyoshikun I've tagged you on the ones that you're best suited to handle. I'll handle the rest.

• [x] update -ls and -lf flag names to -s and -t
• [x] @edyoshikun remove config from estimate-source-to-target-affine
• [x] remove saving of .svgs from estimate-source-to-target-affine
• [x] update optimize-affine docs, specify -o is YAML (edit: this is specified in the example docstring, so I think it's good)
• [x] optimize-affine flags for display and channel choice
• [x] @edyoshikun remove pre_affine_90degree... everywhere, handle output_shape parameter
• [x] add -v flag for optimize-affine
• [x] @edyoshikun add quick comment about how to move to scipy for GPU/cupy (not needed now)
• [x] comment on landscape-mode results. Can napari's transpose provide the rotations we need without changing data storage conventions?

[talonchandler]

• [x] match keys in affine_transform

[talonchandler]

I've run all of the individual steps, and I understand how everything works. Early next week I'll run this on Soorya's data from hummingbird to test it further, and I'll give the code a line-by-line look.

@edyoshikun feel free to comment on my changes if you'd like to take a look.

[talonchandler]

@ieivanov

we should rotate the volumes by 90 deg such that they are in landscape mode rather than portrait mode (i.e. with larger width than height)

@edyoshikun and I discussed this and we'll suggest using napari's rotate feature for landscape (maybe even triggering rotation if Y>X in some scripts). If we want to change the convention for the y axis, we think that would be a larger change that would break convention (at least wrt DaXi and Sapoznik papers). We're open to more discussion.

[ieivanov]

Quick comment: I found the source and target nomenclature quite confusing. Instead, registration algorithms (including ants) use fixed and moving names to refer to the two images being registered. We should switch to that naming

[ieivanov]

In 91375274c06779ddd76561b7e46c1fbafd60ddf4 I've tried to develop what I think is a bit more intuitive manual registration workflow. Here are the highlights:

• The napari viewer doesn't pop up until we are done gathering info on the command line
• The target (fixed) image is plotted as the first layer and the source (moving) image is plotted as an upper layer. Information on these two datasets is also collected in this order
• The source (moving) and target (fixed) images are overlayed by default. The user can toggle between overlay and side-by-side view using the napari grid button.
• At the end, the registered source (moving) image is plotted as an additional layer can can be compared against the target (fixed) image at layer 0

Talon and Ed, let me know what you think

[ieivanov]

input_shape_zyx should be part of RegistrationSettings as it is needed for the LIR cropping

[ieivanov]

A few observations after working through the registration pipeline:

• How apply_affine handles missing data - currently we end up with an image that has both zeros and nans, which displays in napari as a black rectangle on a gray background. Instead, if the input data does not exist, apply_affine should carry on and not modified the empty zarr created at the beginning.
• Registration and cropping should be merged. This is currently creating a lot of friction, requiring multiple copies of the data, and several scripts that need to be run. Just like the empty overhangs after deskew are not useful, the empty regions after registration are not useful either. The ideal workflow would be to supply the two input datasets to be registered, be guided through manual or ants-optimized registration (with option to abort if it doesn't look good), and end up with one registered and cropped multichannel datasets. Let's see how we can make this modular and easy to distribute the workload. I think the recOrder model of reconstruct breaking down into compute-ft and apply-inv-tf would be useful here as well. The overarching CLI could be mantis register - what would be the individual components?
• I have a feeling that the LIR library is an overkill here, but I don't have a better solution either. Our rectangles are always convex - no inward dents, and the transformation after registration is not very drastic, so the solution should be some combination of min/max of the vertices, no? I guess it's good to have a general solution, it just surprised me that it takes a whole minute roughly to find a solution. EDIT: maybe this: https://github.com/planetlabs/maxrect? Seems simpler and better suited for our needs? We want a rectangle that's aligned with the XYZ axes

[edyoshikun]

• Not sure how we should handle those cases. These nan outputs I believe are a product from reconstruction on a blank/zeroes frame. I added a np.nan_to_num(zyx_data, nan=0) prior to applying the transform to a volume. This would convert the nans to zeros, but I am trying to think of the cases in which we don't want this to happen.
• Merged the lir crop into apply_affine. It didn't make sense to separate them since the lir crop has to apply the affine transform anyways to know what the rectangle's shape is. Perhaps my implementation of find_lir() is not the fastest and cleanest. I added a mode so we can plot the output with a red rectangle for debugging.
• I'll test this library quickly and see if it helps.

[ieivanov]

@edyoshikun can you revert the last two commits and add them on a branch off of this one? The registration is currently working and useful, we should be careful not to break it.