search

imagej / imagej2

Open scientific N-dimensional image processing :microscope: :sparkler:
https://imagej.net/
BSD 2-Clause "Simplified" License
1.18k stars 333 forks source link

images larger than 2^29 pixels do not open/load properly #238

Open vasarhelyi opened 5 years ago

vasarhelyi commented 5 years ago

If I want to open an image larger than 2^29 pixels (with File->Import->TIFF Virtual Stack), it does not display properly, there is probably some overflow error at this point as the bottom part of the image exactly after 2^29 pixels is messed up. To illustrate what is the result, I created a triangle of 30000x30000 pixels and opened it with Fiji. See attached images what happens.

example_as_it_should_look

example_opened_with_fiji

vasarhelyi commented 5 years ago

Note: it is the same bug for imageJ and Fiji, tested under Windows10

ctrueden commented 5 years ago

Thanks for the report, @vasarhelyi. The TIFF logic of ImageJ is complicated, so I'm working on verifying this issue in a way that removes TIFF from the picture, first. This will also have the advantage of not needing to pass around a huge example file for testing. Stay tuned.

ctrueden commented 5 years ago

I made a Groovy script to create a very large image from a function. But it became very slow to generate for large images, since Groovy is interpreted. So I recast it as a Java command that can be executed from the Script Editor. Find both versions here:

https://gist.github.com/ctrueden/fcdf93f0aef4db290f982d8df2e90b39

I then generated some very large images:

And all work properly in the interface.

@vasarhelyi This leads me to believe it might be an issue with the TIFF file you generated. The TIFF format is complex, and has some limitations due to byte offsets using 32-bit values internally (unless you use BigTIFF).

How did you generate your TIFF? Can you share it somewhere? Does it read properly when opened from other software? Which other software? What about File › Import › Bio-Formats and/or File › Import › Image...? These latter two commands use different TIFF reader implementations than ImageJ's File › Open..., so worth a try.

vasarhelyi commented 5 years ago

OK, so the process is this:

0) I generate a small image in Windows Paint 1) then resize it in IrfanView to large size and save it as .tif. 2) then I open it in different viewers with the following results:

I uploaded the example I used now here: https://drive.google.com/open?id=1de7IQkf9NJvPFIHeuHuWenidDEYjL4kt

vasarhelyi commented 5 years ago

further trials:

ctrueden commented 5 years ago

Based on stack dumps of the File > Import > Image... approach, it looks like it's very slow—and maybe stuck—doing the LZW decompression:

    at io.scif.io.RandomAccessInputStream.getFilePointer(RandomAccessInputStream.java:160)
    at io.scif.codec.LZWCodec.decompress(LZWCodec.java:409)
    at io.scif.codec.AbstractCodec.decompress(AbstractCodec.java:167)
    at io.scif.formats.tiff.TiffCompression.decompress(TiffCompression.java:272)
    at io.scif.formats.tiff.TiffParser.getTile(TiffParser.java:704)
    at io.scif.formats.tiff.TiffParser.getSamples(TiffParser.java:895)
    at io.scif.formats.tiff.TiffParser.getSamples(TiffParser.java:745)

And later, for substantially longer:

    at io.scif.formats.tiff.DefaultTiffService.undifference(DefaultTiffService.java:96)
    at io.scif.formats.tiff.TiffParser.getTile(TiffParser.java:705)
    at io.scif.formats.tiff.TiffParser.getSamples(TiffParser.java:895)
    at io.scif.formats.tiff.TiffParser.getSamples(TiffParser.java:745)

The decompression logic is single-threaded; I'm seeing 100% CPU used while doing this over several minutes.

Note that the io.scif (SCIFIO) code here was forked from loci.formats (Bio-Formats), and testing Bio-Formats's bfconvert command line tool, I'm seeing precisely the same issue.

I tried converting x.tif to x2.tif using the bfconvert command line tool (bfconvert x.tif x2.tif), but gave up after ~20 minutes of it crunching on the undifference step.

Maybe there's a bug creating an infinite loop. Or maybe the byte decoding logic is just really slow. I'll dig a little deeper.

OrlandoGrigo commented 5 years ago

I have excaclty the same issue with large images. Is there any fix for this issue?

ctrueden commented 5 years ago

@OrlandoGrigo Can you be more specific? Are you also using TIFF? If so, are you using LZW compression? If you are not sure: which tool are you using to create the TIFF files?

OrlandoGrigo commented 5 years ago

@ctrueden Thank you for your reply. I generate the images from images acquisitions using python. The images i am generating are JPG, but i have tried converting them to TIFF.

I am not using the LZW compression.

In order to convert JPG to TIFF I was using GIMP

ctrueden commented 5 years ago

@OrlandoGrigo I tried replicating your workflow by creating a 46000 x 46001 image in ImageJ, exporting it to JPEG, opening in GIMP, and finally exporting to TIFF. But I do not have enough RAM on my laptop to successfully complete all these steps. I'm afraid I won't be able to dig more deeply into this issue unless/until I have easy access to a workstation with lots of RAM.

For now, my main thought is to try different settings when exporting your TIFF. Which compression scheme used (if any) may affect whether/how this bug manifests.

DavidStirling commented 4 years ago

I can confirm that I'm also encountering this issue, both with compressed and uncompressed .tif files. After loading the 2^29th pixel ImageJ (FIJI) appears to give up and load an all-white image before crashing. Importing with BioFormats on the other hand takes a long time, but loads up to around the 2^29th pixel before filling the rest of the canvas with nonsense/repeated sections of the original image. If I crop the images down to below that size limit they load without issue.

ctrueden commented 4 years ago

I didn't realize that @vasarhelyi also reported this on the Image.sc Forum here:

https://forum.image.sc/t/error-importing-tiff-virtual-stack-with-size-over-2-29-pixels/25494

vasarhelyi commented 4 years ago

The given bug report link says that "This bug is not available." Any news on what happened? It would be nice to fix this functionality... Thanks, G.

2019 November 1, Péntek 17:47 CET dátummal, Curtis Rueden notifications@github.com ezt írta:

I didn't realize that @vasarhelyi also reported this on the Image.sc Forum here:

https://forum.image.sc/t/error-importing-tiff-virtual-stack-with-size-over-2-29-pixels/25494

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/imagej/imagej/issues/238#issuecomment-548860589

ctrueden commented 4 years ago

@rasband Do you know if the bug you filed was ever accepted?

rasband commented 4 years ago

The bug report was accepted. This is the message I got from Oracle:

Dear Java Developer,

We have finished evaluating your report and have assigned it a Bug ID: JDK-8223943. The issue is visible on bugs.java.com at the following url JDK-8223943 (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8223943).

To provide more information about this issue, click here.

We work to resolve the issues that are submitted to us according to their impact to the community as a whole, and make no promises as to the time or release in which a bug might be fixed. If this issue has a significant impact on your project you may want to consider using one of the technical support offerings available at Oracle Support.

Regards, Java Developer Support

ctrueden commented 4 years ago

@vasarhelyi have you had a chance to try with Java 13, as suggested in the forum thread?

BoudewijnvanLangerak commented 4 years ago

Is there any news on this subject? We are experiencing similar problems and are willing to contribute to the project. Are there any plans to upgrade the java version?

rasband commented 4 years ago

Have you found a version of Java that does not have this problem?

BoudewijnvanLangerak commented 4 years ago

Just tested with java 13 and java 11 both seem to work. I can open an image of 2.8 Gb. The same image I can not open running on java 8. So it really seems to be a java issue. I would like to help to upgrade the java version if you need the help. Please let me know.

BoudewijnvanLangerak commented 4 years ago

I have build local version on java 11. How should we proceed?

papalagirauscher commented 3 years ago

Hello! As reported, opening very large tiff files without any firlefanz like hyperstack and distortions works using Java version 11 as well as 15. I was not able to make it run using Java 13 or 8 which is delivered with Fiji/ImageJ on default.

How to make imagej work with different java versions (as of today and on linux):

  1. Download a nojre version of fiji
  2. Download the launcher from https://imagej.net/Launcher (for me 5.0.4-20200905.044936 worked, but not the normal one delivered with Fiji) and replace the launcher in the fiji.app (change file name to blabla64 as in fiji.app)
  3. Download an openjdk (for me 11 and 15 worked) and unpack it
  4. Either add export JAVA_HOME="/path until INSIDE of the openjdk" export PATH=$JAVA_HOME/bin:$PATH to your bashrc, or always start Fiji like this --> /PATHTOFIJI/Fiji.app/ImageJ-linux64 --java-home \ path until INSIDE of the openjdk

You are welcome 😁

ctrueden commented 3 years ago

@papalagirauscher Thanks for the instructions. There is also this FAQ entry:

https://imagej.net/Frequently_Asked_Questions#How_do_I_launch_ImageJ_with_a_different_version_of_Java.3F

Your instructions work, but you can alternately just rename the Fiji.app\java folder to something like Fiji.app\java.old and then install Java 11 system-wide, and the launcher should pick up on it.

Why didn't the shipped launcher work? What happens? We certainly want to make sure that the launcher shipping with Fiji works with all versions of the JDK. CC @hinerm.

papalagirauscher commented 3 years ago

Hello! Yes, the problem for me, as an example, is that I cant change the system wide JAVA due to workplace restrictions. This easier approach was therefore not possible for me. The shipped launcher threw errors like crazy when I tried to use different JAVA versions with it --> xingcg not found. I am aware that xincgc is not part of java over version 8. I therefore concluded that the launcher does not recognize the newer jdk. Luckily 5.0.4 worked. I am however a Linux dummy and cant rule out wrong usage, but wanted to provide my solution.

Warm regards, Jan

ctrueden commented 3 years ago

@papalagirauscher Thanks for the additional details.

We recently updated the native launcher code to support discovery of OpenJDK bundles that were unpacked as-is directly beneath the java/linux-amd64 (or whatever platform). So you could for example download the Zulu OpenJDK 11 and unpack the archive inside Fiji, moving aside the old Java 8 that comes bundled, and the next time you launch Fiji it should use that new version of Java. Unfortunately, every time we ship this new snazzier native launcher, someone somewhere has a problem with their Fiji no longer launching, so at the moment I believe the native launcher is rolled back to the pre-snazzy version that doesn't work so well in this regard. @hinerm correct me if I'm wrong. It's a thorny problem.

papalagirauscher commented 3 years ago

Hello! Thats a great information! - maybe it should be added to the FAQ? https://imagej.net/Frequently_Asked_Questions It would save less tech savvy people a few headaches and a night shift 😁 I actually tried this, replacing the jdk folder in the fiji.app with an openjdk 11 bundle. It did however not work and I ended up with the same error that I reported above. I am however aware that it is prob impossible to create a solution that runs out of the box for everybody.

Thank you for the reply, I really appreciate the ImageJ/Fiji project and would not be able to do my research without it. Keep on the good work!

Warm regards, jan

hinerm commented 3 years ago

so at the moment I believe the native launcher is rolled back to the pre-snazzy version that doesn't work so well in this regard.

@ctrueden that is correct. @papalagirauscher there is a thread on the forum for testing the new launcher. You can add the update site https://sites.imagej.net/Launcher-6-test/ to try the latest launcher.