search

AllenCellModeling / aicsimageio

Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in Python
https://allencellmodeling.github.io/aicsimageio
Other
202 stars 51 forks source link

Mosaic merge of LIF #278

Closed psobolewskiPhD closed 1 year ago

psobolewskiPhD commented 3 years ago

System and Software

Description

Reading a mosaic merge from LIF generates a mismatched image. It appears that the image is, for example, instead of x=4 y=3 merged as y=4 x=3 Example LIF scene "R1" (TileScan 2/B/2/R1) https://www.dropbox.com/s/jk9se8i1kpzqvsn/20210428_24w_L929_Ho_B2C3.lif?dl=0 For comparison, scene "R1_Merged" (TileScan 2/B/2/R1_Merged) is the mosaic merge by LAS X Note: I'm not sure the overlap is taken into account?

Expected Behavior

Generated a proper merge automatically or provide an option to provide the merging pattern and overlap

Reproduction

Using: https://www.dropbox.com/s/jk9se8i1kpzqvsn/20210428_24w_L929_Ho_B2C3.lif?dl=0

from aicsimageio import AICSImage
img = AICSImage("20210428/20210428_24w_L929_Ho_B2C3.lif")
img.set_scene("R1")
img.dims
img.set_scene("R1_Merged")
img.dims

Environment

As mentioned above, I have native Apple Silicon M1 arm64 python and aicsimageio

psobolewskiPhD commented 3 years ago

I've figured out the problem—I think. I'm going to outline my observations and thought process for future reference for anyone interested—I'm not yet sure how to really do this in python and integrate a fix into AICSImageIO.

Anyhow, as far as I can tell the current approach in AICSImageIO doesn't really use the information from readlif and makes some assumptions which aren't always met. Here's how I understand it: readlif returns mosaic_positions which is a list of tuples: (FieldX, FieldY, PosX, PosY)

What AICSImageIO does is figure out the number of columns and rows to get the basic mosaic shape (ny, nx): https://github.com/AllenCellModeling/aicsimageio/blob/57e83755d8beb1dbbb880a4d3b4657770812d36d/aicsimageio/readers/lif_reader.py#L581-L586

where: last_tile_position = selected_scene.info["mosaic_position"][-1] BUT: This works only if the last position of mosaic_position, the last tile imaged, is really at the ny, nx position. For a zig-zag tiling (the default in our LAS X) this may not be the case for an even number of rows. See this sequence: (0,0), (1, 0), (2, 0), (3, 0), (3,1), (2,1), (1,1), (0,1) which would set last_tile_position to the tile (0,1) and return wrong ny, nx values. Let's add another row to make it an odd number: (0, 2), (1, 2), (2, 2), (3,2) Now, last_tile_position is the correct tile and we get (ny, nx) as (4, 3). We have 4 cols, 3 rows, so that matches.

BUT: When stitching, the dimensions of the mosaic end up wrong, because I think row/col are mixed up here: https://github.com/AllenCellModeling/aicsimageio/blob/57e83755d8beb1dbbb880a4d3b4657770812d36d/aicsimageio/readers/lif_reader.py#L537-L541

the range of ny is 4, the number of columns (obtained from FieldX in mosaic_position), while nx is the number of rows (obtained from FieldY). So for the example above, the mosaic prepared by AICSImageIO ends up as 3 col by 4 row. For my (x:1920, y:1440) images and a 4 col, 3 row mosaic, I get square mosaics from AICSImageIO stitching: (YX):(5757, 5758) which corresponds to: (y:1440x4=5760, x:1920x3=5760). LAS X stitching of the same tiles:(Y,X)(3899, 6811) vs. the expected: (y:1440x3 = 4320, x:1920x4=7680). This reflects the overlap in tiles, more on this later.

Further, when placing each tile in the mosaic, in other words selecting the image from the list indexed by M and putting it in a given (col, row), M is computed as: (row_i * nx) + col_i Again this will only work for L-R, row-wise and not zig-zag. For the example above this yields: M: [0, 1, 2, 3, (3+3=6), (3+2=5),(3+1=4), (3+0=3)]

So to solve this, I think it's probably best to get the mosaic_position list from readlif and iterate over that, because the index of the list is the proper M index for that X, Y tile position. Also, one should be able to use the physical coordinates PosX and PosY to account for tile overlap. First, use the values of the PosX, PosY of the (0,0) tile to get the origin and make the other tile Pos be relative values to that origin. One can rescale back to pixels to make life easier. Then the position delta between tiles relative to the tile size will reflect the overlap. First pass, one could just replace the pixels in the overlap region with those from the next tile, but some blending could potentially also be implemented in the future to mimic what LAS X does (see: https://imagej.net/plugins/image-stitching).

evamaxfield commented 3 years ago

Yea, we need to do a pass on all of our mosaic stitching for CZI and LIF again.

Fully agree that we should use the bounding box info and not just the indices.

BrianWhitneyAI commented 1 year ago

We have added a new maintenance feature to clean up stale issues and PRs. Adding this comment to set a baseline for ‘Stale’

SeanLeRoy commented 1 year ago

Hi @psobolewskiPhD, I opened #480 to provide a potential avenue to solve this issue. I agree with you that this package is swapping the X and Y coordinates by default and that seemed to have messed with your image. However, even after turning that off I seemed to be unable to match up with the expected image, so I enabled flipping the X and Y coordinates (which results in a 180 degree rotation) too which seemed to make it line up. If you get the opportunity I'd appreciate you taking a look at my pull request to see if that aligns with your thinking and may solve this issue.

I did not address improving the stitching of overlapping tiles using the physical coordinates. Is that something that would need to be fixed to consider this ticket done?

psobolewskiPhD commented 1 year ago

Sorry, been busy in RL. To be clear, as far as I was able to tell (https://github.com/AllenCellModeling/aicsimageio/issues/278#issuecomment-922492259) it's not just that X and Y were being swapped but that tile positions were based on a faulty assumption of row-by-row scanning. Anyhow, I don't think that a solution to his issue needs to deal with overlap. Just returning rough mosaics that have the tiles place properly would be a good fix—sometimes overlap may be really low or even 0. I think the key is handling the actual tile locations in a way that accounts for zig-zag acquisition which is the default on our Leica 'scope.

I think that cropping tiles to deal with overlap certainly could/would be a potential followup?

PS. I will try and take a closer look at the code/PR ASAP, but don't let me block you!

SeanLeRoy commented 1 year ago

@psobolewskiPhD Okay sounds great! This was closed as part of my PR being merged, but feel free to re-open or create another ticket if you find the fix unsatisfactory for your use case!