search

JaneliaSciComp / BigStitcher-Spark

Running compute-intense parts of BigStitcher distributed
BSD 2-Clause "Simplified" License
18 stars 10 forks source link

computational complexity of big sticher fusion shows undesirable scaling behaviour with number of tiles #1

Closed VolkerH closed 1 year ago

VolkerH commented 2 years ago

Hi @StephanPreibisch ,

thanks for this new project. Need to set up Spark first, but keen to give this a try. Saw the announcement on twitter but as I don't have a twitter account I'll ask a related question here:

We ran into issues running fusion with a large number of 2D tiles (not using the Spark version). The fusion step would just take many hours when fusing around 700 individual 2D tiles (mosaic scan of a whole slide). We observed that the scaling behaviour with the number of tiles was very unfortunate (polynomic), where I would expect it should only grow approximately linearly with the number of output pixels.

As I had the impression that Big Stiticher was primarily developed for Light-Sheet data (fewer but much larger volume tiles and not many 2D tiles) this scaling behaviour with the number of tiles might have gone unnoticed?

EDIT to add:

The above behaviour was noticed on the non-Spark version of affine fusion, any chance this has already been fixed with this code?

StephanPreibisch commented 2 years ago

Hi, yes, that issue should be fixed ... but I haven't tried 2D images yet ...

VolkerH commented 2 years ago

Thanks for the reply, that is great to hear.

Is the fix for the complexity issue with the number of tiles fixed only available this Spark-based version or is it (will it) also in the "normal" Big Stitcher Fiji plugin fusion code? Still hoping I can get around setting up Spark to try this out.

StephanPreibisch commented 1 year ago

The normal BigStitcher code is now also scaling much better with the number of tiles ...