Array size too large

[atarkowska]

Following fixes in https://github.com/openmicroscopy/bioformats/pull/3257 latest BF-6.0.0-SNAPSHOT built based on https://github.com/olatarkowska/bio-formats-build/tree/develop_merge_trigger cause following error (file opens without issues with BF-5.9.2)

(Fiji Is Just) ImageJ 2.0.0-rc-69/1.52i; Java 1.8.0_172 [64-bit]; Mac OS X 10.14.2; 77MB of 8866MB (<1%)

java.lang.IllegalArgumentException: Array size too large: 165240 x 165240 x 1 x 2
    at loci.common.DataTools.safeMultiply32(DataTools.java:1286)
    at loci.common.DataTools.allocate(DataTools.java:1259)
    at loci.formats.TileStitcher.openBytes(TileStitcher.java:119)
    at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
    at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)

[melissalinkert]

Is this data available somewhere for testing? Probably the first thing to determine is if this is a problem with the number of series, XY dimensions of each series, or XY positions of each series.

If you uncheck the Stitch tiles option when opening the dataset, are the series counts and image dimensions the same in 5.9.2 and 6.0.0-SNAPSHOT? If so, the positions are likely the problem. Checking the Display OME-XML metadata option should show all of the reported XY positions, which should make it easier to see any changes between the two versions.

[atarkowska]

Indeed in 5.9.2 there is no position that was added in 6.0.0 <WellSample ID="WellSample:0:0:0" Index="0" PositionX="-6.9" PositionXUnit="m" PositionY="-6.0" PositionYUnit="m">

[sbesson]

The dimensions reported by the exception i.e. 165240 x 165240 x 1 x 2 certainly make me suspect the well sample position population added in #3257 now allows well samples to be stiching together.

As suggested in https://github.com/openmicroscopy/bioformats/issues/3314#issuecomment-459086197, I see two options depending on the desired workflow:

• disable the basic tile stitching from Bio-Formats by unchecking Stitch tiles. This should treat all series separate
• if tile stitching is a requirement, then you might need to define a smaller region to open similarly to any large XY image e.g. using the Crop option

[atarkowska]

My major question here is why it says 165240 x 165240 I think that is not correct, see https://openmicroscopy.slack.com/archives/CDHCGMMC1/p1540363102000100 they are 23Kx40K

[melissalinkert]

165240 x 165240 is definitely incorrect; each image is 1080 x 1080, so this implies 153 x 153 tiles (when there are 153 total fields to be stitched).

The problem occurs with the OrientationMatrix, which for all fields is:

x	y	z	a
1.002630	0.004879	0	-6.9
0.004618	-1.003768	0	6.0
0	0	1	0

This means that no two fields have the same X or Y position, so TileStitcher sets the tile count to 153 x 153:

https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/TileStitcher.java#L280

TileStitcher really isn't meant for doing any kind of sophisticated stitching, so this isn't too surprising. Some potential options:

1. use something meant for proper stitching instead of TileStitcher
2. add an option to TileStitcher to ignore differences in position values below some (configurable?) threshold (have not tried this, not sure how well it would work)

3. add an option to OperettaReader to normalize the orientation matrix so that only values of 0, 1, and -1 are allowed in the first 3 columns. In this case, the matrix used would be:

   x	y	z	a
   1	0	0	-6.9
   0	-1	0	6.0
   0	0	1	0

Option 3 seems to work OK in local tests. Ignoring the orientation matrix entirely is probably not viable, as this will mix up the order of fields in Y.