paulscherrerinstitute / ch.psi.imagej.hdf5

ImageJ HDF5 Plugin
13 stars 5 forks source link

Error when opening files with "Virtual stack" disabled #9

Open MarkRivers opened 1 year ago

MarkRivers commented 1 year ago

I need to open an HDF5 file as a regular stack, not a virtual stack. The reason is that the dataset is signed 16-bit, so to view it correctly in ImageJ I need to apply a calibration using the following macro:

run("XOR...", "value=1000000000000000 stack");
run("Calibrate...", "function=[Straight Line] unit=[Gray Value] text1=[0 65535] text2=[-32768 32767 ]");
run("Enhance Contrast", "saturated=0.5");

This works fine when I open signed 16-bit netCDF files with the netCDF plugin because it reads them as a "real" stack, rather than a virtual stack. The calibration does not work when reading HDF5 files into a virtual stack, because virtual stacks are read-only.

When I uncheck the "Virtual stack" box in the dialog window for this plugin I get an error message in the main ImageJ window saying there is an error reading the file.

Is this checkbox actually supported so that reading into regular stacks is allowed? If so, how do I figure out what the error is?

MarkRivers commented 1 year ago

I took a quick look at the code, and it seems that in my case with virtual stack disabled and a 3-D dataset it should be executing the code here: https://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/blob/e37c671c6e9ecbe6f2f99fe3c387bef424e7b098/src/main/java/ch/psi/imagej/hdf5/HDF5Reader.java#L317

It seems like an exception is happening, and I am seeing the error message "Error while opening file [filename]." How do I see the logger output to determine what the exception was?

rcatwood commented 1 year ago

Hi Mark, this sounds similar to the problem I was describing last week,

Dec 02, 2022 12:50:24 PM ch.psi.imagej.hdf5.HDF5Reader open WARNING: Error while opening: /mpc/homes/fische_r/NAS/test2.h5 java.lang.Exception: Failed to read scalar dataset: Invalid int size at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:820)

Which goes a bit beyond my Java HDF library knowledge, I could not find how to interpret this specific error message. But it seems to take effect at around a 2^32 limit, regardless of 3d or 4d data dimensions, something is just not data-type to Long which should be? ??

A larger dataset prompts a similar error (now with a h5 file from h5py):

Dec 02, 2022 12:48:00 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: Using manual selection Dec 02, 2022 12:48:02 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: Reading dataset: data Dimensions: 4 Type: 16-bit unsigned integer Dec 02, 2022 12:48:02 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: 4D Image (HyperVolume) Dec 02, 2022 12:50:21 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: Using manual selection Dec 02, 2022 12:50:24 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: Reading dataset: data Dimensions: 4 Type: 16-bit unsigned integer Dec 02, 2022 12:50:24 PM ch.psi.imagej.hdf5.HDF5Reader open INFO: 4D Image (HyperVolume) Dec 02, 2022 12:50:24 PM ch.psi.imagej.hdf5.HDF5Reader open WARNING: Error while opening: /mpc/homes/fische_r/NAS/test2.h5 java.lang.Exception: Failed to read scalar dataset: Invalid int size at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:820) at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:221) at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:51) at ch.psi.imagej.hdf5.HDF5Reader.run(HDF5Reader.java:36) at ij.IJ.runUserPlugIn(IJ.java:237) at ij.IJ.runPlugIn(IJ.java:203) at ij.Executer.runCommand(Executer.java:152) at ij.Executer.run(Executer.java:70) at java.lang.Thread.run(Thread.java:750) Caused by: hdf.hdf5lib.exceptions.HDF5Exception: Invalid int size at hdf.object.h5.H5Utils.getTotalSelectedSpacePoints(H5Utils.java:118) at hdf.object.h5.H5ScalarDS.scalarDatasetCommonIO(H5ScalarDS.java:902) at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:816) ... 8 more

I am testing with one of my "smaller" samples: (867, 849, 2016, 9) which is about 27 GB, max. number of time steps are 50. This a nice size to handle in python, even better with appropiate chunking. I had the idea to load a hyperstack to do some preliminary inspection in Fiji allowing me to slide in time and one spatial direction, I think this would be very useful for users. But maybe Fiji with h5/nc is not the right tool for that and using macros to create a Hyperstack from the original reconstruction output is the better way. No idea how efficiently this would work. Thanks for your help and time.

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/8#issuecomment-1335144267, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESVMPTIIPND7QS55TDLWLHRCFANCNFSM6AAAAAASRX4LUM. You are receiving this because you commented.Message ID: @.**@.>>

From: Mark Rivers @.> Sent: 04 December 2022 00:07 To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Subscribed @.***> Subject: Re: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

I took a quick look at the code, and it seems that in my case with virtual stack disabled and a 3-D dataset it should be executing the code here: https://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/blob/e37c671c6e9ecbe6f2f99fe3c387bef424e7b098/src/main/java/ch/psi/imagej/hdf5/HDF5Reader.java#L317

It seems like an exception is happening, and I am seeing the error message "Error while opening file [filename]." How do I see the logger output to determine what the exception was?

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9#issuecomment-1336275212, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESUTOJ73IXW5AX2GF6LWLPHCLANCNFSM6AAAAAASS7KGHY. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

rcatwood commented 1 year ago

Hi Mark .. asking as another user .. what is the size of the image you are reading? Is it bigger than 2^32 or 2^31 voxels / data elements?

Is it the line 331 int startIdx = lev * size ;

this would get bigger than the intmax of 32 bit integer would it not, if the java is a bit like ‘C’ ? is it a limitation of ImageJ 2 itself ?? But I can open a big stack virtually and then ‘duplicate’ the whole thing into a real stack , so it can’t be a limitation in ImageJ2 Fiji , can it ?? can this simply be ‘long int’ ? do you know how to change and build it just to test ? (I don’t think I do ..)

From: Mark Rivers @.> Sent: 03 December 2022 23:48 To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Subscribed @.***> Subject: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

I need to open an HDF5 file as a regular stack, not a virtual stack. The reason is that the dataset is signed 16-bit, so to view it correctly in ImageJ I need to apply a calibration using the following macro:

run("XOR...", "value=1000000000000000 stack");

run("Calibrate...", "function=[Straight Line] unit=[Gray Value] text1=[0 65535] text2=[-32768 32767 ]");

run("Enhance Contrast", "saturated=0.5");

This works fine when I open signed 16-bit netCDF files with the netCDF plugin because it reads them as a "real" stack, rather than a virtual stack. The calibration does not work when reading HDF5 files into a virtual stack, because virtual stacks are read-only.

When I uncheck the "Virtual stack" box in the dialog window for this plugin I get an error message in the main ImageJ window saying there is an error reading the file.

Is this checkbox actually supported so that reading into regular stacks is allowed? If so, how do I figure out what the error is?

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESSLQXECFAHBT36COM3WLPE23ANCNFSM6AAAAAASS7KGHY. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

MarkRivers commented 1 year ago

@rcatwood thanks for the hints. In ImageJ I could not see the logger output, but on Fiji I could. I am indeed getting the same error as you.

Dec 04, 2022 8:15:26 AM ch.psi.imagej.hdf5.HDF5Reader open
WARNING: Error while opening: C:\Data\Czas\DIA0000NB_A_recon.h5
java.lang.Exception: Failed to read scalar dataset: Invalid int size
    at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:820)
    at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:325)
    at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:51)
    at ch.psi.imagej.hdf5.HDF5Reader.run(HDF5Reader.java:36)
    at ij.IJ.runUserPlugIn(IJ.java:239)
    at ij.IJ.runPlugIn(IJ.java:205)
    at ij.Executer.runCommand(Executer.java:152)
    at ij.Executer.run(Executer.java:70)
    at java.lang.Thread.run(Thread.java:748)
Caused by: hdf.hdf5lib.exceptions.HDF5Exception: Invalid int size
    at hdf.object.h5.H5Utils.getTotalSelectedSpacePoints(H5Utils.java:118)
    at hdf.object.h5.H5ScalarDS.scalarDatasetCommonIO(H5ScalarDS.java:902)
    at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:816)
    ... 8 more

My datasets are 16-bit integers. I tested creating and opening datasets of different sizes:

The size of these arrays is not a problem with ImageJ/FIJI. I can read netCDF files with datasets of 16 GB into real stacks in ImageJ with no problem. This appears to me to be a bug in the HDF5 Java library. Perhaps it is fixed in a newer version?

rcatwood commented 1 year ago

I and another user were barking up a wrong tree; thinking it had to do with 4d arrays since ialways had Virtual ticked— but virtual seems to have no effect in 4d ; i can understand that, but it exposed this problem (sooner since 4d gets big pretty fast !)

Sent from Outlook for iOShttps://aka.ms/o0ukef


From: Mark Rivers @.> Sent: Sunday, December 4, 2022 2:39:12 PM To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Atwood, Robert (DLSLtd,RAL,SCI) @.>; Mention @.> Subject: Re: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

@rcatwoodhttps://github.com/rcatwood thanks for the hints. In ImageJ I could not see the logger output, but on Fiji I could. I am indeed getting the same error as you.

Dec 04, 2022 8:15:26 AM ch.psi.imagej.hdf5.HDF5Reader open WARNING: Error while opening: C:\Data\Czas\DIA0000NB_A_recon.h5 java.lang.Exception: Failed to read scalar dataset: Invalid int size at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:820) at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:325) at ch.psi.imagej.hdf5.HDF5Reader.open(HDF5Reader.java:51) at ch.psi.imagej.hdf5.HDF5Reader.run(HDF5Reader.java:36) at ij.IJ.runUserPlugIn(IJ.java:239) at ij.IJ.runPlugIn(IJ.java:205) at ij.Executer.runCommand(Executer.java:152) at ij.Executer.run(Executer.java:70) at java.lang.Thread.run(Thread.java:748) Caused by: hdf.hdf5lib.exceptions.HDF5Exception: Invalid int size at hdf.object.h5.H5Utils.getTotalSelectedSpacePoints(H5Utils.java:118) at hdf.object.h5.H5ScalarDS.scalarDatasetCommonIO(H5ScalarDS.java:902) at hdf.object.h5.H5ScalarDS.read(H5ScalarDS.java:816) ... 8 more

My datasets are 16-bit integers. I tested creating and opening datasets of different sizes:

The size of these arrays is not a problem with ImageJ/FIJI. I can read netCDF files with datasets of 16 GB into real stacks in ImageJ with no problem. This appears to me to be a bug in the HDF5 Java library. Perhaps it is fixed in a newer version?

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9#issuecomment-1336428736, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESWLWJBHUDFLKN2YTGTWLSUJBANCNFSM6AAAAAASS7KGHY. You are receiving this because you were mentioned.Message ID: @.***>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

MarkRivers commented 1 year ago

@rcatwood I found that you are correct that the problem is with the number of array elements, not the size of the array in bytes. I did a test with 32-bit floating point arrays.

So the limit is not 4 GB it is 2^31 array elements.

MarkRivers commented 1 year ago

I just tested a fresh install of Fiji with the default HDF5 reader, not the PSI version. It has the same issue. It can read 1024x1024x2047 datasets, but fails with 1024z1024z2048 datasets. When it fails I get this message: image

rcatwood commented 1 year ago

I always had that with the default .. i think one is limited by number of bytes while the other is by number of data elements ! But virtual stack , then ‘duplicate ‘ for 3d files seems to work around (but isn’t a fix!)

Sent from Outlook for iOShttps://aka.ms/o0ukef


From: Mark Rivers @.> Sent: Sunday, December 4, 2022 5:05:20 PM To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Atwood, Robert (DLSLtd,RAL,SCI) @.>; Mention @.> Subject: Re: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

I just tested a fresh install of Fiji with the default HDF5 reader, not the PSI version. It has the same issue. It can read 1024x1024x2047 datasets, but fails with 1024z1024z2048 datasets. When it fails I get this message: [image]https://user-images.githubusercontent.com/6115534/205504822-a42186a0-5d41-4976-8947-1738cfdabdb0.png

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9#issuecomment-1336468514, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESRSZDHYMGLQKUEPWJDWLTFNBANCNFSM6AAAAAASS7KGHY. You are receiving this because you were mentioned.Message ID: @.***>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

MarkRivers commented 1 year ago

I always had that with the default .. i think one is limited by number of bytes while the other is by number of data elements !

I just tested and both versions of the HDF5 plugin are limited by the number of elements, not the number of bytes. The following fail with both plugins:

However, if the last dimension is changed to 2047 then all files load as real stacks in both plugins.

But virtual stack , then ‘duplicate ‘ for 3d files seems to work around (but isn’t a fix!)

Yes, that works. But it temporarily doubles the amount of memory needed which is not good.

rcatwood commented 1 year ago

Interesting .. i did not test it as systematically. You think it's in the HDF library itself ? But using h5py i haven't come across trouble opening big 3d images ( apart from slowness )

Sent from Outlook for iOShttps://aka.ms/o0ukef


From: Mark Rivers @.> Sent: Sunday, December 4, 2022 7:16:46 PM To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Atwood, Robert (DLSLtd,RAL,SCI) @.>; Mention @.> Subject: Re: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

I always had that with the default .. i think one is limited by number of bytes while the other is by number of data elements !

I just tested and both versions of the HDF5 plugin are limited by the number of elements, not the number of bytes. The following fail with both plugins:

However, if the last dimension is changed to 2047 then all files load as real stacks in both plugins.

But virtual stack , then ‘duplicate ‘ for 3d files seems to work around (but isn’t a fix!)

Yes, that works. But it temporarily doubles the amount of memory needed which is not good.

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9#issuecomment-1336495352, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESTDPRZUBH7EYC73H5DWLTUZ5ANCNFSM6AAAAAASS7KGHY. You are receiving this because you were mentioned.Message ID: @.***>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

MarkRivers commented 1 year ago

You think it's in the HDF library itself ? But using h5py i haven't come across trouble opening big 3d images ( apart from slowness )

The problem is not in the native HDF5 library, so C and Python (which wraps the C library) work fine. The problem is in the Java implementation of HDF5. I believe the PSI version of the HDF5 plugin uses a native Java implementation of HDF5 which was itself written at PSI. I don’t know what Java library the original HDF5 plugin uses.

There is a Java library provided by the HDF Group itself. That is not a native Java implementation, it is a Java wrapper around the C library. I would be surprised if that version has this issue.

But using h5py i haven't come across trouble opening big 3d images ( apart from slowness )

I'm surprised that you say slowness for h5py. I have been really impressed with the speed of h5py. On my system I can read an 8 GB (1920, 1200, 1800, Uint16) HDF5 data file in 4 seconds from my SSD drive. That is 2 GB/s.

rcatwood commented 1 year ago

From: Mark Rivers @.> Sent: 04 December 2022 20:33 To: paulscherrerinstitute/ch.psi.imagej.hdf5 @.> Cc: Atwood, Robert (DLSLtd,RAL,SCI) @.>; Mention @.> Subject: Re: [paulscherrerinstitute/ch.psi.imagej.hdf5] Error when opening files with "Virtual stack" disabled (Issue #9)

You think it's in the HDF library itself ? But using h5py i haven't come across trouble opening big 3d images ( apart from slowness )

The problem is not in the native HDF5 library, so C and Python (which wraps the C library) work fine. The problem is in the Java implementation of HDF5. I believe the PSI version of the HDF5 plugin uses a native Java implementation of HDF5 which was itself written at PSI. I don’t know what Java library the original HDF5 plugin uses.

There is a Java library provided by the HDF Group itself. That is not a native Java implementation, it is a Java wrapper around the C library. I would be surprised if that version has this issue.

But using h5py i haven't come across trouble opening big 3d images ( apart from slowness )

I'm surprised that you say slowness for h5py. I have been really impressed with the speed of h5py. On my system I can read an 8 GB (1920, 1200, 1800, Uint16) HDF5 data file in 4 seconds from my SSD drive. That is 2 GB/s.

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/issues/9#issuecomment-1336510963, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4TESQUGUG5F644JI7L5K3WLT5ZNANCNFSM6AAAAAASS7KGHY. You are receiving this because you were mentioned.Message ID: @.**@.>>

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

MarkRivers commented 1 year ago

Java implementation of HDF5 which was itself written at PSI

I think I misspoke, I believe the Java HDF5 implementation is from ETH not PSI.