Ome.tiff file size limit

[spimager]

Hi All,

We need to acquire images far beyond the 4GB tiff file size limit. As I understand, this is not currently possible. I was planning on customising our version of the SPIMAcquisition code to save the stacks as single separate tiffs which we can re-stack inside of FIJI on our servers, similar to what is currently possible in standard MicroManager.

I just wanted to see what you all thought of the idea before I set out and if you could potentially recommend anything?

Thanks for the amazing software so far!

Gabriel

[dscho]

We need to acquire images far beyond the 4GB tiff file size limit. As I understand, this is not currently possible.

It should be possible by inserting writer.setBigTiff(true); in or around https://github.com/openspim/SPIMAcquisition/blob/6a6cca5630facfb40144c53ad3f079d24d789298/src/main/java/spim/io/OMETIFFHandler.java#L107

[spimager]

Thanks a lot! However, when I try compile with Maven, I get the "Cannot find symbol" error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3. 1:compile (default-compile) on project SPIMAcquisition: Compilation failure [ERROR] /C:/Users/OpenSPIM/Dropbox/workspace/SPIMAcquisition/src/main/java/spim/ io/OMETIFFHandler.java:[107,31] cannot find symbol [ERROR] symbol: method setBigTiff(boolean) [ERROR] location: variable writer of type loci.formats.IFormatWriter

[pawlowska]

the "Multi-D. Acq" of Micro-Manager uses another solution: if stack is bigger than 4 GB, it gets divided, first part is saved as 3.99 GB file and the second part as the rest. Such a solution could be incorporated into the plugin, people who acquire smaller stacks would not notice a difference

EDIT: @gabrieldavisjones it seems that it's not so easy as @dscho suggested. Currently writer is of class ImageWriter that desn't have the setbigtiff method. This method is available eg in OMETiffWriter http://downloads.openmicroscopy.org/bio-formats/5.0.4/api/loci/formats/ImageWriter.html http://downloads.openmicroscopy.org/bio-formats/5.1.2/api/loci/formats/out/OMETiffWriter.html

[dscho]

@pawlowska excellent analysis! I do not have time to take this further, but maybe you want to start from 13c68aa4fdf51e3cff73479a497230284543368d?

[pawlowska]

I tried today to see what happens if I just use OMETiffWriter, but I didn't know how to add the dependency correctly (i.e. to do what you did in pom.xml @dscho). From tomorrow I'm on vacation, I'll try again next week, or maybe @gabrieldavisjones has time earlier.

[ctrueden]

You can call the OmeTiffWriter API on an ImageWriter as follows:

public void enableBigTiff(final ImageWriter w) {
  final OmeTiffWriter otw = w.getWriter(OmeTiffWriter.class);
  otw.setBigTiff(true);
}

In your pom.xml, you'll need to add:

<dependency>
  <groupId>ome</groupId>
  <artifactId>formats-bsd</artifactId>
</dependency>

[dscho]

@ctrueden have you looked at https://github.com/openspim/SPIMAcquisition/commit/13c68aa4fdf51e3cff73479a497230284543368d? I could imagine that you did not have the time, because you almost recapitulated the same changes...

[spimager]

@ctrueden Just made the 13c68aa changes and it's working perfectly! Thanks a lot.

[dscho]

@gabrieldavisjones thanks for the feedback. Now, what you could do to drive this forward is to craft a nice commit message, mentioning that you tested this (and how), and open a proper Pull Request for @hkmoon to merge.

[ctrueden]

@dscho Actually, I did look at 13c68aa4fdf51e3cff73479a497230284543368d. It is different: it calls ImageWriter#getWriter(String) which quasi-initializes the given file (without calling setId... weird...).

The call to ImageWriter#getWriter(Class), on the other hand, does not do any initialization (since it doesn't know the id string yet). I was thinking this fact would be really important because the setBigTiff(boolean) method requires that setId has not yet been called on the writer object.

But since @gabrieldavisjones says that 13c68aa4fdf51e3cff73479a497230284543368d is working for him as-is, I guess I was wrong and my approach is not needed. :ok:

[pawlowska]

I tested the https://github.com/openspim/SPIMAcquisition/tree/big-tiff as it is now. It is able to acquire a file bigger than 4 GB, but at the same time, when I acquire a small file, say 100 MB, it is now saved differently - previously Fiji was opening such files automatically, now the "Bio-formats Import Options" window pops up. With the old files, it was even possible to open them in the Windows built-in image viewer, now it is not possible even with the small file. This looks like an unnecesarry complication.

@dscho, could you explain what the if (writer instanceof OMETiffWriter) does? I added a line ij.IJ.log("calling setBigTiff(true)"); after line 109 to understand what is happening and it looks like now setBigTiff(true) is called always, and this causes the files to be unreadable by "normal" viewers even if they are small. (BTW, is there a way to see in a tiff file whether it has the bigtiff tag?)

[pawlowska]

If I understand this correctly https://trac.openmicroscopy.org/ome/changeset/6decd24c801c753183a156f2d716ccd3d69fdc49/bioformats.git from the newest version of Bioformats setting bigtiff happens automatically when setid is called. But from the code it looks like it's not called at all, or am I missing something?