Corrupted few columns of CZI file with 4.0.0

boazmohar commented 3 years ago

System and Software

aicsimageio Version: 4.0.0
Python Version: 3.8.10
Operating System: linux [Oracle Linux Server release 8.3]

Description

Corrupted few columns of czi data.

Expected Behavior

Loading it with CziFile or native software shows there is no corruption in the raw data

Reproduction

import aicsimageio
print(aicsimageio.__version__)
>>4.0.0
from aicsimageio import AICSImage
img = AICSImage('/nrs/svoboda/moharb/ExM/20210606_B6_IHC1/IHC1/IHC1.czi', chunk_by_dims=['CZ'])
print(img.shape)
>>(1, 4, 2170, 1920, 1920)
dask_array = img.dask_data

plt.imshow(xarray[0,0,1,:,:], vmax=1000)

Environment

aicspylibczi-3.0.1

I can upload the file to dropbox but it is 64Gb (lmk). Thanks!

evamaxfield commented 3 years ago

Hey! Thanks for the bug report! Hopefully we can fix it soon.

By CziFile you mean Gohlke's CziFile library? (we have a CziFile object in our reader which is why I need the clarification)

And yes please share the file if you can.

evamaxfield commented 3 years ago

Also, just wanted to say that I see you are using chunk_by_dims, we changed that to just chunk_dims in 4.0.

evamaxfield commented 3 years ago

Ignore my last statement. I see that CziReader still uses chunk_by_dims

boazmohar commented 3 years ago

Hi,

Yes this is the library
I found this still happens with a smaller file.

I think it is looping the right side of the image to the left side. These were captured on a recently updated Z1 light-sheet, if that helps.

evamaxfield commented 3 years ago

Thanks for the smaller test file. We have had problems with light sheet images before... https://github.com/AllenCellModeling/aicspylibczi/issues/81

I will try to put some time at this tonight or Thursday night.

boazmohar commented 3 years ago

@JacksonMaxfield Thanks! I read up on the issue you linked and I wanted to clarify that my data doesn't have "raw" and "deskewed" as it is not from the Lattice microscope (I think I only have "raw").

evamaxfield commented 3 years ago

Small update. Seems this issue is down at the aicspylibczi level (our library for reading CZI files) which means it's likely somewhere in the C++ code.

boazmohar commented 3 years ago

Hi @JacksonMaxfield, thanks for the update. Would you like me to open an issue there?

Boaz

evamaxfield commented 3 years ago

Nope no worries. I will simply transfer this issue over to that repo if needed but leaving it here until next dev meeting which will be on Monday.

evamaxfield commented 3 years ago

A couple of updates.

I am going to transfer this issue to the aicspylibczi repo as it seems to be an issue on that side.
I personally don't have a timeline for resolution but I think @toloudis may have time to look at it? The general statement is, we will keep tracking this issue and post updates.

Sorry for the long process.

toloudis commented 3 years ago

I'm starting to take a look today. Will see how far I get.

toloudis commented 3 years ago

Here are my findings after digging into aicspylibczi and libCZI for a day. I'm using the smaller Overview.czi file to test.

the image loads correctly with czifile, and it also loads correctly in Zen
open source libCZI (as compiled into aicspylibczi) appears to load the pixel data with a bad file offset, and so the data is shifted in the way shown in the pictures above
earlier versions of aicspylibczi show the same problem
I observed the same problem on both Mac and Windows. The error seems to clearly have to do with file offsets where the data is being read
I have a one line change in libCZI that seems to fix this image, but I have a feeling it may not be robust against ALL cases and would need to be reviewed by someone at Zeiss who knows that library

One question I would have is: how was the image generated? Are we seeing an image literally as it came off the microscope and/or has it been edited in any way? I'm not sure if the inconsistent offsets I see in the file data are allowed and reasonable, or if there is some other bug happening way upstream of all this.

@sebi06 I wonder if you might have some insight into this.

The one line change is in libCZI/Src/libCZI/CZIParse.cpp, line 252: https://github.com/zeiss-microscopy/libCZI/blob/master/Src/libCZI/CziParse.cpp#L252 I add:

    offset = subBlckSegment.data.entryDV.FilePosition;

which overrides the given offset with the filePosition contained in the variable length subblock directory entry.

boazmohar commented 3 years ago

@toloudis Thank for looking into this! I can only add as to the generation of the data which is right off the microscope (Zeiss Z1 lightsheet). No edits were made. If there is a complied version of this change that I can pull, I can test it with other files I generated off the same microscope.

sebi06 commented 3 years ago

I need to point out that LightSheet Z1 is still running the old ZEN Black SW. I would not create any big changes bases on such CZI image files.

Ideally they work as well but I would rather only test against CZI from ZEN Blue.

The overview.czi was acquired by me a while ago, but I cannot rember anymore, if I did some postprocessing like stitching etc.

But I can provide fresh CZI (using the latest software version) at any time. Those should be the reference.

What would you need?

ptahmose commented 3 years ago

The PR https://github.com/ptahmose/libCZI/pull/1 should fix the issue discussed here. Note that the fix is applied to this repository (in the branch 'wolframlink') only at this point in time. Hope this fixes the problem, and thanks for the analysis.

toloudis commented 3 years ago

@ptahmose @pthamose @JuergenBohl Thank you for this fast fix! We have a compile error because of mismatched types in the call to max which I patched like this: https://github.com/AllenCellModeling/libCZI/commit/2a8fdeb2f38d4bfae287b3987883118ab2b05938

evamaxfield commented 3 years ago

This was marked as resolved but will not be actually fixed until we push a new version. Which will be shortly!

evamaxfield commented 3 years ago

This should now be resolved. Can upgrade and use with pip install --upgrade or pip install aicspylibczi==3.0.2

AllenCellModeling / aicspylibczi