Fiji closes when trying to open BigStitcher-Spark generated .zarr

[clairemitchell308]

Hello

I tried to use this function to open a .zarr file I had generated using BigStitcher-Spark on a Linux cluster. Fiji closed with the following error:

terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
Aborted (core dumped)

Does this plugin only work for .zarr files generated using PetaKit5D or should it work for all .zarr files (can you suggest a test file I can try?)? I'm not very au fait with .zarr or running Fiji on our cluster so I apologise in advance if it's something very obvious. I can try to upload the .zarr somewhere if that would help but it's a few hundred GB.

Many thanks Claire

[MattHazard]

Hello.

Could you share the .zarray metadata file with us here? We support .zarr files created by other software but it is possible that your file may have some missing metadata. Also how much RAM do you have in your system? The entire file will be loaded to RAM and it may use more RAM than the size of the image when loading it.

If we are still unable to solve the issue then I'd be happy to have a look at the image if you are able to share the whole .zarr with us.

Best,

Matthew Mueller

[clairemitchell308]

Hi Matthew,

Thanks a lot for the quick reply. I don't see a .zarray on the .zarrs I'm generating so maybe that's the main issue?

I stitched and exported the demo file provided with BigStitcher-Spark as it's much smaller than the .zarrs I'm generating to upload here (zipped by me, not BigStitcher). I have the same issue with Fiji closing when I try to drag and drop this into Parallel Fiji Visualiser so I don't believe it's a RAM issue (I am running this on an interactive HPC node with 990 GB RAM.

fused.zarr.zip

Many thanks Claire

[MattHazard]

Hi Claire,

Thank you for providing this to me. The .zarray file may be hidden by default on your OS or file browser when you are looking at the zarr files. The file you provided appears to be empty other than metadata but I believe I understand the issue. BigStitcher-Spark writes out zarr files in a .zgroup which we don't directly support but you should still be able to open the zarr files within the .zgroup. In your case, the actual zarr files are fused.zarr/ch0/s0 and fused.zarr/ch1/s0 as these are the folders that contain the .zarray files that we use to open the zarr file. For this plugin, we currently require the zarr file to have the .zarr extension so you may have to rename the folder to s0.zarr for our plugin to read it when you drag it in. I will work to remove that requirement for the next release as I think the only real requirement for a zarr file is the existence of the .zarray metadata file in the folder.

Best,

Matthew Mueller

[clairemitchell308]

Hi Matt

Whoops, sorry for sending over an empty one! I tried your suggestion of renaming the ch0/s0 folder to have a .zarr extension but still see Fiji crashing. I also tried on my Windows machine in case it was a Fiji/HPC/Linux problem but I have the same issue so I think it must be the files themselves. I've attached the s1 folder which is small enough to fit within the file size limit but can also upload the full stitched dataset somewhere else if that would be helpful?

s1.zarr.zip

Thanks a lot and best wishes Claire

[MattHazard]

Hi Claire,

Thank you for providing me with the image. I was able to replicate your issue and I have figured out what the problem was. A bug in the function we call from cpp-zarr prevented big-endian (> in your dtype metadata) from being read as all our images had been little-endian. I have fixed the issue and can now open the file you provided with the plugin. I will have this fix in the next release, but for now I can provide you with the fix for the Linux version. You can replace the Parallel_Fiji_Visualizer_lib/libcppZarr.so file with the one in this tar.gz: libcppZarr.tar.gz. Let me know if this fixes the issue for you.

Best,

Matthew Mueller

[clairemitchell308]

Ah yes that's fixed the problem thank you very much! I've also tried on my microscope data which I stitched using BigStitcher Spark but get a slight issue with the intensities of the pixels in the opened image? From previous experience I'm wondering if it's a signed/unsigned issue? (I also realise this isn't the main purpose of your plugin so if you want to tell me to go and use another importer that's fine :D ). When I import using the HDF5/N5/Zarr/OME-NGFF importer it looks correct (but is much slower than your nice plugin!).

I've attached a small example, they also open with a different dimension order than the other importer but this is easily fixed by reslicing the stack. s6.zarr.zip

Thanks a lot! Claire

[MattHazard]

Hi Claire,

The issue was actually with the endianness. Since we are using little-endian systems, the byte order of the pixel values needs to be swapped from big-endian. I've added support for this type of thing now. Here's the fix: libcppZarr.tar.gz. This will also be available in the next release. Since this is part of the zarr spec, I have no problem adding support for it. I was under the impression that big-endian systems are quite rare nowadays. If you don't mind me asking, what are the computers that wrote out these zarr files?

As for the dimension order, this is expected as zarr doesn't provide a standard for arranging the dimensions. It seems BigStitcher uses zyx like Python now but we used yxz for our plugin since most of our image processing pipeline is in Matlab where yxz is the standard.

Let me know if all your issues are resolved now.

Best,

Matthew Mueller

[clairemitchell308]

Hi Matthew

Sorry for the delay, yes that looks perfect to me thank you! I asked our HPC admin and he said as it's Ubuntu it should be little-endian so we're not really sure what happened to be honest but your fix has solved it.

Many thanks and best wishes Claire

[MattHazard]

No problem. I'm happy I could resolve this for you. Feel free to open another issue if you have any other questions or problems.