search

PreibischLab / multiview-reconstruction

Software for the reconstruction of multi-view microscopic acquisitions like Selective Plane Illumination Microscopy (SPIM) Data
GNU General Public License v2.0
22 stars 22 forks source link

Fusion using ExportN5API does not preserve multi-resolutions #47

Closed CellKai closed 1 year ago

CellKai commented 1 year ago

issue description

When fusing a multi-resolution dataset (h5/xml) with the N5API to e.g. BDV-compatible HDF5, the resulting fused h5/xml only contains a single resolution level, see below screenshots from inspect both dataset in Fiji using Plugins › HDF5 › Load HDF5 File...

potential downstream issues

Viewing "Big" fused image (e.g. > ~100 tiles) in BDV is terribly laggy / not feasible

reproduce the issue

download the demo dataset and run below IJ-macro code, then inspect both dataset in Fiji using Plugins › HDF5 › Load HDF5 File...

run("Fuse dataset ...", "select=/home/kai/Data/work/bigstitcher/Grid_3d_h5_aligned/grid-3d-stitched-h5/dataset.xml process_angle=[All angles] process_channel=[All channels] process_illumination=[All illuminations] process_tile=[All tiles] process_timepoint=[All Timepoints] bounding_box=[All Views] downsampling=1 interpolation=[Linear Interpolation] pixel_type=[16-bit unsigned integer] interest_points_for_non_rigid=[-= Disable Non-Rigid =-] blend preserve_original produce=[Each timepoint & channel] fused_image=[ZARR/N5/HDF5 export using N5-API] define_input=[Auto-load from input data (values shown below)] export=HDF5 create hdf5_file=/home/kai/Data/work/bigstitcher/Grid_3d_h5_aligned/grid-3d-stitched-h5/fused/fused.h5 xml_output_file=/home/kai/Data/work/bigstitcher/Grid_3d_h5_aligned/grid-3d-stitched-h5/fused/fused.xml");

Crosspost

https://forum.image.sc/t/bigstitcher-n5-api-fused-n5-hdf5-has-no-pyramids-and-does-not-display-well-in-bdv/80925

StephanPreibisch commented 1 year ago

Hi, I just implemented the support to create multi-resolution outputs for fused HDF5/N5/ZARR images. Simply update Fiji and check “create multi-resolution pyramid” during fusion (see screenshot).

Please let me know if that works (it does for me) …

Thanks so much, Stephan

https://github.com/PreibischLab/multiview-reconstruction/commit/e5eb4764c3b7bfa1c2f3605d489af4d469d88b82

Screen Shot 2023-07-19 at 5 38 14 PM
CellKai commented 1 year ago

Hi @StephanPreibisch ,

thank you for the update!

I downloaded a vanilla Fiji (2023-07-20, win64), activated only the BigStitcher update site, and fused some of my data (1 angle, 3 channels, 1 illumination, 100 tiles, 1 timepoints, 0 missing views) to hdf5/xml using the N5-API.

I can confirm that:

So thats really fantastic :star_struck: !!

However I noticed another issue (apologies :sweat_smile: )

Opening the small demo dataset works seemingly fine, but when I open my own "big" h5/xml dataset (1 angle, 3 channels, 1 illumination, 100 tiles, 12 timepoints, 0 missing views, ~400 GB, registered but not fused, on Windows)

In older versions of BigStitcher/mvr/bdv (Using spimdata version: 0.9-revision, Using multiview-reconstruction version: 1.2.9, bdv jar6.2.2), this was much faster.

So somehow there seems to be a performance decrease to open/display the dataset, but I can't tell with certainty in which version of BigStitcher/mvr/bdv it was introduced.

crosspost

https://forum.image.sc/t/bigstitcher-n5-api-fused-n5-hdf5-has-no-pyramids-and-does-not-display-well-in-bdv/80925/4

StephanPreibisch commented 1 year ago

Hi, is there any chance you can share this dataset with me? Then I could debug what is going on ... I do not have such delays ... maybe it's OS-related or it's the dataset.

CellKai commented 1 year ago

Hi, I uploaded : T2_sect2_h5_tp0.zip (38.1 GB) : https://filesender.switch.ch/filesender2/?s=download&token=06cdbadb-406f-4e52-aafc-11175bac322a

I truncated the dataset (original: 1 angle, 3 channels, 1 illumination, 100 tiles, 12 timepoints, ~400 GB, registered but not fused) and included only timepoint 0, the other partitions are missing on purpose so the upload is smaller (38 GB). It still reproduces the delay-issue for me on Windows:

StephanPreibisch commented 1 year ago

Great, thanks so much, will debug next week!

StephanPreibisch commented 1 year ago

I just downloaded it and it opens in about 5-10 sec for me in my up-to-date Fiji … do you load it via a network connection maybe?

StephanPreibisch commented 1 year ago

In any case I'll have a look what it does in these 10 secs ... might be revealing. Unfortunately, I have to do performance reviews this week, so only next week seems reasonable at this point

CellKai commented 1 year ago

Hi @StephanPreibisch , thank you again for all the testing!

it opens in about 5-10 sec for me in my up-to-date Fiji … do you load it via a network connection maybe?

Thats a good point, I load indeed from network storage.

It is our common use case. In our facility we provide our user with 60+ virtual machines that all use a common network storage.

I tested a bit more, and

independent of the delay issue, I noticed that

Thank you again for your help, its much appreciated :smiley:

CellKai commented 1 year ago

@StephanPreibisch it seems that the delay is a BDV issue and has hence been moved to https://github.com/bigdataviewer/bigdataviewer-core/issues/165

Concerning BigStitcher, this then leaves only the last point

when closing the BigSitcher Window, the “select dataset for stitching explorer” Dialogue window opens again. It used not to be the case in previous versions.

StephanPreibisch commented 1 year ago

that's actually on purpose :)

StephanPreibisch commented 1 year ago

I found this much better, since often I do want to open another dataset ... if people hate it I am happy to add some sort of checkbox for it ...

CellKai commented 1 year ago

that's actually on purpose :)

actually, after using it now for a couple of times, I liked it as well :)