Display of very large 3D images shows striped artefacts at some zoom-in view levels

[kgabor]

Opening an n5 dataset of one (or multiple) very large 3D tile(s) in BDV (open xml/hdf5) or BigStitcher, the displayed image shows striped artefacts and/or disintegrate (rectangular pieces misplaced), usually on the right side of the display in the default XY plane (Z-axis) view.

The artefacts seem to be consistent, i.e. in a given angle and zoom view it's consistently the same. The artefacts disappear and the displayed image is correct if the view is zoomed out to an "overview" level when the image occupies only a fraction of the display window. Or if zoomed-in to a "detailed view" when the display window is a fraction of the whole image.

If multiple of such 3D tiles are opened by BigStitcher, the patterns appear on each tile.

The problem can be reproduced with the attached constant image (value = 1). Dimensions: 14208 x 10688 x 12032, uint16. Single resolution. The .xz.zip archive inflates to a (still gzip compressed) n5 storage of about 17GB.

The data is displayed correctly in N5-viewer, however.

bigdataviewer version: 6.2.2

exaspim_dummy_tile.tar.xz.zip

[tpietzsch]

bigdataviewer version: 6.2.2 Where does that number come from? bigdataviewer-core version 6.2.2 doesn't exist. Do you mean 6.0.2. I couldn't reproduce it running the current bdv master from an IDE? Could you try that? (Just run this main method https://github.com/bigdataviewer/bigdataviewer-core/blob/e29a9253ce0de30d9bcb76d6a623c0f8d5265a45/src/main/java/bdv/BigDataViewer.java#L776-L778 and set the fn to your dataset.)

Trying to reproduce the error, I was running into OutOfMemory constantly, because the blocks in you dataset are so huge (1 GB per block). With many concurrent loading threads that also all need temporary memory for decompression I was hitting my limit of (admittedly tiny) 16GB RAM constantly. So I had to reduce number of threads etc before I could see anything, but then I couldn't find artefacts.

Could you try to:

• make screenshots of the artefacts you are seeing
• run from IDE and see whether OutOfMemory or any other exceptions occur
• make another similar dummy dataset with smaller block sizes (e.g. 64x64x64) for testing

[kgabor]

Where does that number come from? bigdataviewer-core version 6.2.2 doesn't exist. Do you mean 6.0.2.

(base) jupyter@7262e9376515:~/Fiji.app/plugins$ ls big*
bigdataviewer_fiji-6.2.2.jar

I also encountered the OutOfMemory exception but not in case of the real data. Indeed exploding decompression with multithreading seems to easily hit the OutOfMemory exception. I was able to get an initial view of the dummy with about 130GB mem that showed the artifacts without hitting loading exceptions. I will try to make a better dummy dataset with smaller block sizes and multiple resolutions as well (for faster display).

Zoomed in or zoomed out, the display looks ok:

At the default zoom, it does not:

[kgabor]

The dummy dataset seems to behave a bit differently, it is always broken regardless of zooming. Maybe connected to the (lack of) lower resolutions (in the dummy). Also, I made a screenshot rotated into the YZ plane at a high X value. It hit the OutOfMem exception but seems to have completed/retried the load. (Loading the dummy is worse than loading real data, it is painfully slow and CPU intensive due to decompression; one display takes 5 minutes w/ 130GB) rendering is 10+ minutes w/ 130GB mem.)

Default and zoomed out XY plane views:

Rotated, got exception but load seems complete:

[mkitti]

(base) jupyter@7262e9376515:~/Fiji.app/plugins$ ls big*\nbigdataviewer_fiji-6.2.2.jar

Ah you mean bigdataviewer-fiji: https://github.com/bigdataviewer/bigdataviewer_fiji

[kgabor]

I encountered two scenarios when the problem reproduces and another one that seems ok. It may be simply a bigdataviewer-core old version problem or a bad combination of module versions.

Produces patterns:

A maven environment copy-pasted from intelliJ, see attached classpath file.

java -Dfile.encoding=UTF-8 -classpath $(cat bigstitcher_dev_classpath.txt) net.preibisch.stitcher.plugin.BigStitcher

bigstitcher_dev_classpath.txt

Does not produce patterns, seems to display the same data correctly:

This is a Fiji download without JRE, it uses the same jdk as the one above. I replaced the jars of imglib2-algorithm to 0.13 and multiview-reconstruction to the lazygauss build in this run which may be irrelevant for this bug.

~/Fiji.app/ImageJ-linux64 -Dimagej.updater.disableAutocheck=true

fiji_working_lsR.txt

Please findls -lRlisting attached.