Fast_Translation_Fusion runs out of memory

[MichaelSNelson]

Our group is trying to stitch whole slide images (larger than Java array size limit) using BigStitcher, but we are having problems with time and/or memory scalability:

Time performance. BigStitcher takes a very long time due to an N^2 issue when running on many tiles. (Details at ctrueden/tasks#11)

Memory performance. So, we tried using the experimental Fast_Translation_Fusion plugin (adding it to plugins.config locally), but we run out of memory due to use of CellImg for all input and output images.

This issue here is about the latter: memory performance of Fast_Translation_Fusion. Here is what we have learned so far in our investigation:

• Fast_Translation_Fusion hits memory caps very quickly even at modest total file sizes.
• The initial cause of the memory errors appears to be loading the entire image into memory using CellImgFactory calls, and replacing instances of that with DiskCachedCellImgFactory does bypass that, although we are not confident that this change is best from an algorithmic standpoint. Original OOM error stack trace passed through Fast_Translation_Fusion.java. Result, still an OOM error.
• Using heap dumps and a memory analyzer, @ctrueden was able to track another possibility in that the DownsampleTools.openAndDownsample() call of Fast_Translation_Fusion.java, defined on line 449 of DownsampleTools.java had a boolean openCompletely that might avoid this error if changed from true to false. Changed the OOM error line to FastFusionTools. This is part of the addTranslated() function: “add input image to output image with given offset”
• Data set size we are testing with is 8.85GB, what is the expected memory capacity needed for such a data set? With Fast_Translation_Fusion, 24GB RAM is insufficient.
• The resulting image attempts to be displayed using img.show(), causing problems for large 2D images (Java array size) FastFusionTools.java. Possibly swap this to BDVFunctions.show or make the display of the stitched image optional?
• Relatedly: can the resulting image be written out in pyramidal format instead of as a single tiff? (@ctrueden asked in the BDV channel if there was a way to write out ImgLib2 images to the BDV HDF5 format easily, and Christian Tischer linked to this code he wrote. Might be worth a try for us some time soon.)

Do you think any of these problems could be solved on the BigStitcher side? We are now running up against limitations in our knowledge and understanding of the BigStitcher code. With @ctrueden we could potentially dig more, but it is time consuming and we thought we would ask you first here if you know how to address this?

Please let me know if you could use any further information or explanations. I am working on sharing our test data set (the ~9GB of tif files), but wanted to first absolutely verify that it is ok to share :)

Cheers, Mike

[MichaelSNelson]

Adding links to two data sets, converted to composite from RGB:

1. A small 600MB linear string of 24 images with 10% overlap, I used it to determine that the unedited fast_translation_fusion requires 8GB of memory to succeed with this data set, while restricting the JVM to 7GB results in an OOM failure.
2. A larger data set, 360 3000x3000 image with 10% overlap representing the target size image.

In case it helps, links to the test code where the above mentioned changes were made (thanks @hinerm). https://github.com/MichaelSNelson/BigStitcher/tree/fast-fuse-in-menu https://github.com/MichaelSNelson/multiview-reconstruction/tree/disk-cached

[hinerm]

@tpietzsch we were looking at a heap dump and noticed this cyclical relationship in SoftRefLoaderRemoverCache's PhantomReference.. is there a chance this could cause a memory leak with phantom refs being unable to be GC'd?

[tpietzsch]

@hinerm No, that should all be fine. (If it wasn't this would have exploded a long time ago...)

First, cyclic dependencies are not a problem for GC, if the whole cycle is unreachable it will be collected.

Second, the PhantomRef "watches" the value (i.e. Cell in CachedCellImg), not the Entry. This cycle doesn't matter for that, the value is not strongly referenced anywhere in the cycle. When value is GCed, the PhantomRef is enqueud, and the entry is removed here: https://github.com/imglib/imglib2-cache/blob/6c2ec9e37acb8d4f89cc9a34349141bd228381d9/src/main/java/net/imglib2/cache/ref/SoftRefLoaderRemoverCache.java#L284-L300. This makes the whole cycle unreachable.