Porting full FOV stack stitching + surface maping to Mesoscope's splitting module

[nataliaorlova]

To help registering 2P data to ISI, we have been acquiring full FOV stacks on the rigs. This data will capture almost entire cranial window in the same coordinate system as the main physio datasets (surface, depth and time series).

This stack is being reformatted (averaged and stitched) and the surface of the main high resolution data set is pasted into the full FOV. The code that performs this lives off pipeline and users are asked to run a Jupyter Notebook at the ed of each experimental session in order to generate image below that will be uploaded to LIMS.

We would like to request porting this code to the etl pipelines so that the raw data could be uploaded to LIMS and stitching and surface mapping is performed by LIMS.

The code that generate the image below can be found in this notebook: https://github.com/nataliaorlova/meso_tools/blob/master/meso_tools/notebooks/fullfield_stitching.ipynb

[danielsf]

@nataliaorlova Can you point me to an example of the input data for this code that lives on Isilon? It looks like your notebook is pointing to files that live on the D: drive of one of your personal machines.

I can see the output of this process (the fullfield.tiff file) in /allen/programs/mindscope/production/vipssn/prod0/specimen_1175512441/ophys_session_1190324963 but I cannot see the averaged_surface.tiff file (presumably because it was archived...)

[danielsf]

Actually: should this be able to run on any averaged_surface.tiff file? I have some old ones from previous tiff-splitting adventures lying around.

[danielsf]

@nataliaorlova

Sorry, I just noticed that the fullfield.tiff is both an input and an output of your notebook (or... your notebook appends something to fullfield.tiff). I do not have any examples of fullfield.tiff files lying around. Can you point me to some example data on Isilon? Thanks.

[nataliaorlova]

Averaged_surface.tiff indeed gets split and archived in LIMS, and my code does something similar. I copied an input data set for your development here.

[danielsf]

@nataliaorlova

I've finished a first implementation of the utility you've requested. Here is a comparison of what your code generates and what the ophys_etl code generates. Recall that the test data used to generate this figure is here

\\allen\programs\braintv\workgroups\multiphoton\Natalia\fullfield_stitching_data_fro_Scott

There are obviously two big differences

1) Dynamic range. When generating the image, I normalize the background image and the ROIs separately so that their pixels run the full range of np.uint16 (i.e. each ROI is separately normalized to cover [0, 65535]; comparing the dynamic range of one ROI to another/the background is basically meaningless). This results in the background being darker than the ROIs, presumably because of that bright patch at (row, col) ~ (50, 1200) causes the actual field of view in the background to appear quite dark. Is this a useful image, or do we need to do something to make sure that the ROIs and the background have comparable dynamic ranges?

2) Placement of ROIs. I think this is just the reflection of a bug in your implementation. At this line

https://github.com/nataliaorlova/meso_tools/blob/master/meso_tools/stitch_full_field.py#L226

you transpose the coordinates for the origin of your image from (x, y) space to (row, col) space. However, when you use those coordinates here

https://github.com/nataliaorlova/meso_tools/blob/master/meso_tools/stitch_full_field.py#L266-L267

you treat them as if they are still in (x, y) space. I have verified that this transposition is why your implementation puts the ROIs in a different location than my implementation.

If you want to play around with my implementation, install the branch ticket/522/fov/stitching of ophys_etl_pipelines. The image (as a numpy array) is generated with

from ophys_etl.modules.mesoscope_splitting.full_field_utils import (
    stitch_tiff_with_rois)

img = stitch_tiff_with_rois(
            full_field_path=full_field_tiff_path,
            avg_surface_path=averaged_surface_tiff_path)

Let me know what you think of this output (especially the 2 questions above).

Slightly related question: should we be doing something to add a bright border around the ROIs so that people looking at the images can readily see them, or will that be too confusing?

[nataliaorlova]

Hi Scott, thanks a lot for implementing this so quickly.

I'd like to fist mention that the main use for this data in the future automatic processing steps is for ISI-2P registration. In view of that, I think it would be most beneficial to have the used dynamic range for both the FullField tiff, and the mapped into it surface images be similar. This is also why I did not yet implement the bright border around the mapped images.

Now thinking about it, I wonder whether we should save just the stitched Full Field separately for automatic registration. The version with mapped hi-res surface images can be useful for other purposes, figures, visual assessment, etc, and it would be OK to have the mapped images have higher brightness as well as colored frames around the images, and maybe even text denoting which HVA this is (this info is entered during acquisition and saved in the ScanImage metadata, I can alter add where exactly, have no access to tiffs now, it should be something like hSI.rois[1]['name']). Then the second image can be used for actual registration. What do you think?

And yes, the # 2 is a bug, thanks so much for fixing it!

We should try and run it on a few more sample datasets, so thanks to pointing me to how to do that! I'll think about the tests and do it when I'm back.

[matchings]

@danielsf @nataliaorlova I'd like to comment on one of Natalia's statements above - "the main use for this data in the future automatic processing steps is for ISI-2P registration" - as it turns out, we have an additional, rather important, use case for these full FOV images, and that is for ophys-mFISH registration for the learning project.

@cbrrrry recently discovered that he is able to identify vasculature patterns in the reconstructed lightsheet volumes, and mapping the mesoscope full FOV image (which shows plenty of vasculature) with the regions where ophys data was acquired overlaid on them to the lightsheet data would be extremely helpful to know where to look for the cells that were imaged with ophys. Having this information to initialize the registration algorithm simplifies the problem significantly. Without it, it may not be possible to routinely link the ophys and mFISH datasets.

This is all to say that this issue (and https://github.com/AllenInstitute/ophys_etl_pipelines/issues/535) have a much higher priority in my mind than they did previously.

[nataliaorlova]

I've ran the code on three datasets from Meso1 and Meso2 (SIDs 1226257370, 1226714845 and 1212880506) the results are great:

Additionally, we discussed with Scott saving the outputs in an h5 file in the following format:

• fullfield stitched (more raw data, might be more useful for automatic registration)
• fullfield stitched, with hires surfaces mapped into it, frames added around all hires surface regions and possibly text with HVA name (this is more for visualization purposes)
• hires surface images with mapping coordinates

Otherwise, the ticket can be closed

[danielsf]

Addressed by

540

549

556