Support for ImageJ's fake TIF files, when file size is > 4Gb

[bugalo]

I have some tiff stacks bigger than 4G, and TIFFStack is not able to handle them. Are you planning on supporting them?

Thank you!

[DylanMuir]

Hi @bugalo, I need a little more information.

As long as tifflib provided by Matlab supports TIFF files > 4Gb, then TIFFStack should support them out of the box. However, this hasn't been tested.

A couple of things to check: what version of Matlab are you using, and on what platform? With what software were the files generated? Are the files really BigTIFF format files, or are they ImageJs fake TIFF format, that it uses for large files?

When you say TIFFStack can't handle them, exactly what happens?

D.

[bugalo]

Edit: this message has some incorrect information. Read the next ones.

Original message:

Hi Dylan,

thank you very much for responding so quickly.

The Tiff is a stack that I created using ImageJ, and I thought it was using BigTIFF format. I will check it as soon as I am in front of that computer, so maybe that is the problem.

Regarding Matlab, I am using version R2015b on Linux.

When I type Matlab's command

numel( imfinfo( 'my_file.tif' ) )

I get 1, although the stack has 570 planes. However, when I use,

Tiff( 'my_file.tif', 'w8')

I am able to correctly read the stack. For that reason, I assumed it was in the correct BigTIFF format. When I use

size( TIFFStack( 'my_file.tif' ) )

I get 2048x2048, so it is not reading the whole stack.

Anyway, I will recheck everything tomorrow morning when I can access that computer, and see if the image is in the correct BigTIFF format. I will update my response then.

Thank you very much for responding and for TIFFStack, I believe it is going to be very helpful.

[DylanMuir]

Hi,

Yep, that’s the problem. ImageJ makes a fake TIFF file, with a single TIFF-format image as the header, followed by the entire stack as a binary data file. Then they call the whole thing a TIF file. I really don’t know how they justify such a weird design decision.

Anyway, I’m surprised that

Tiff( 'my_file.tif', 'w8’)

works at all. I’ll take a look into it.

Cheers, Dylan.

On 5 Apr 2016, at 8:39 pm, bugalo notifications@github.com wrote:

Hi Dylan,

thank you very much for responding so quickly.

The Tiff is a stack that I created using ImageJ, and I thought it was using BigTIFF format. I will check it as soon as I am in front of that computer, so maybe that is the problem.

Regarding Matlab, I am using version R2015b on Linux.

When I type Matlab's command

numel( imfinfo( 'my_file.tif' ) )

I get 1, although the stack has 570 planes. However, when I use,

Tiff( 'my_file.tif', 'w8')

I am able to correctly read the stack. For that reason, I assumed it was in the correct BigTIFF format. When I use

size( TIFFStack( 'my_file.tif' ) )

I get 2048x2048, so it is not reading the whole stack.

Anyway, I will recheck everything tomorrow morning when I can access that computer, and see if the image is in the correct BigTIFF format. I will update my response then.

Thank you very much for responding and for TIFFStack, I believe it is going to be very helpful.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub

[DylanMuir]

By the way, Tiff('file.tif', 'w8') immediately overwrites the file with a new empty BigTIFF file, so that doesn't help you at all.

[bugalo]

Hi Dylan,

sorry about the confusion. As I said yesterday, I was not in front of the computer and I didn't remember exactly what was happening. I have fairly large stacks, some with around 4.2GB, and the largest one with 4.8GB. All were created exactly the same with ImageJ, and the only one I cannot open is the 4.8GB one.

It turns out I didn't correctly remember the result of Tiff( 'my_file.tif', 'w8’), as you said, it overwrites the TIFF with a new, empty one. It turns out I cannot open the largest TIFF with any of the Matlab's built-in functions, but I can open even the 4.2GB ones.

I agree it has to be the format in which ImageJ saves the large arrays, I will need to find another way of building my stacks. I am really sorry I made you waste your time with this issue, you can already close it.

Thank you.

[DylanMuir]

Hi @bugalo, I have just pushed an update which natively supports ImageJ's fake TIF files > 4Gb. Please let me know if this works for you. Note that you will need the MappedTensor package.

[bugalo]

I've just checked it, and I can confirm that it works.

Thank you very much for taking the time to implement it.

[dickinson-lab]

This is a useful enhancement, but requires calling the function tiffread31_readtags.m, which I found is very slow (see also #28). Commenting out the line

[bIsImageJBigStack, bIsImageJHyperStack, vnStackDims, vnInterleavedIJFrameDims] = IsImageJBigStack(tiffread31_readtags(oStack.TIF_tr31, oStack.HEADER, 1), numel(oStack.HEADER));

and replacing with

bIsImageJBigStack = false;
bIsImageJHyperStack = false;
vnStackDims = [];
vnInterleavedFrameDims = [];

makes TIFFStack run ~20x faster (2017 iMac desktop, MATLAB R2021a). This of course breaks the support for ImageJ fake Tiff files >4GB, but is an improvement for smaller files including those we work with (which are saved from MicroManager 2.0gamma and thus forced to have file size <= 4GB).

Perhaps a future update could allow the user to pass a parameter specifying that the file is an ImageJ fake TIFF, thereby eliminating the need to (slowly) read all the tags in order to figure this out programmatically. Alternatively, Dylan's suggestion to speed up tag reading with a custom MEX file (#28) could help.