search

jries / SMAP

GNU General Public License v3.0
63 stars 20 forks source link

Cannot load tif files over 4 Gb #5

Open xavpinson opened 2 years ago

xavpinson commented 2 years ago

Hi,

I am trying to do some STORM images reconstruction and I am apparently unable to load tif files over 4Gb (smaller images load without any issue). I am pretty incompentent in Matlab but from what I can understand some module responsible for loading tif files seems to be the problem.

I am copying below the output I get when I try to open a tif file over 4Gb.

Any help/idea/suggestion would be very much appreciated.

Thanks!

Xavier

main directory: D:\Home\evpinto\Desktop settings directory: E:\Programmes\SMAP\application\settings plugin help directory: E:\Programmes\SMAP\application\Documentation\help D:\Home\evpinto\Desktop

err =

MException with properties:

identifier: 'MATLAB:FileIO:InvalidFid'
   message: 'Invalid file identifier. Use fopen to generate a valid file identifier.'
     cause: {}
     stack: [3x1 struct]
Correction: []

fid =

-1

could not write file E:\Programmes\SMAP\application\Documentation\pdf\SMAP_UserGuide.pdf locally. bioformats package not found. Please select path to bioformats_package.jar in the Preferences. you can download the Matalb toolbox for bioformats at https://www.openmicroscopy.org/bio-formats/downloads/ Error using bfGetReader (line 51) Missing Bio-Formats library. Either add bioformats_package.jar to the static Java path or add it to the Matlab path.

Error in imageloaderOME/openi (line 18)

Error in interfaces.imageloaderSMAP/open (line 224)

Error in interfaces.imageloaderSMAP (line 37)

Error in imageloaderOME (line 14)

Error in imageloaderAll (line 42)

Error in WorkflowModules.Loaders.TifLoader/addFile (line 193)

Error in WorkflowModules.Loaders.TifLoader>loadtif_callback (line 292) simple tiff loader loader no description found could not determine number of images. Directly evaluate file, might take time. Camera not recognized. Use default Error using bfGetReader (line 51) Missing Bio-Formats library. Either add bioformats_package.jar to the static Java path or add it to the Matlab path.

Error in imageloaderOME/openi (line 18)

Error in interfaces.imageloaderSMAP/open (line 224)

Error in interfaces.imageloaderSMAP (line 37)

Error in imageloaderOME (line 14)

Error in imageloaderAll (line 42)

Error in getRoiTif (line 9)

Error in gettif (line 6)

Error in File.Load.Loader_tif>loadfile (line 82)

Error in File.Load.Loader_tif/load (line 15)

Error in File.Load.Loader_auto>loadfile (line 69)

Error in File.Load.Loader_auto/load (line 17)

Error in gui.GuiFile>loadfiles (line 333)

Error in gui.GuiFile/loadbutton_callback (line 145)

Error in gui.GuiFile>@(varargin)obj.loadbutton_callback(varargin{:}) (line 408) simple tiff loader loader could not find camera file in global settings. Using default file. Warning: Inputs must be character vectors, cell arrays of character vectors, or string arrays.

In findsettingsfile (line 9) In interfaces/imageloaderSMAP/getmetadatacam (line 179) In interfaces/imageloaderSMAP/getmetadata (line 160) In imageloaderTifSimple/openi (line 20) In interfaces/imageloaderSMAP/open (line 224) In interfaces/imageloaderSMAP (line 37) In imageloaderTifSimple (line 11) In imageloaderAll (line 47) In getRoiTif (line 9) In gettif (line 6) In File.Load.Loader_tif>loadfile (line 82) In File.Load/Loader_tif/load (line 15) In File.Load.Loader_auto>loadfile (line 69) In File.Load/Loader_auto/load (line 17) In gui.GuiFile>loadfiles (line 333) In gui/GuiFile/loadbutton_callback (line 145) In gui.GuiFile>@(varargin)obj.loadbutton_callback(varargin{:}) (line 408) In gui/GuiMainSMAP/makeGui (line 353) In SMAP (line 23)

camera calibration file settings/camera.mat not found.

err =

MException with properties:

identifier: 'MATLAB:badsubscript'
   message: 'Index in position 2 exceeds array bounds.'
     cause: {}
     stack: [9x1 struct]
Correction: []
Rieslab commented 2 years ago

Hi Xavier, From the error messages it seems that SMAP was using the TifSimple.m loader, because the other loaders (Micromanager loader or OME loader) could not read the file. In there I use the Matlab Tiff class. In general, I thought that tiff files were restricted to <4GB, that’s why Micromanager saves data in chunks of 4GB. The first errors denote that you did not properly give write access to all SMAP folders (have a look at the installation instructions). The real error indeed was in the loading of the file, meaning that Matlab using this Tiff class could not open it properly. If you suggest me an alternative (sufficiently fast) Matlab code, I would be happy to write a tiff loader for larger files. You could also fix the write access problem and select the correct path for the bioformats.jar. Maybe the OME reader can read larger tiff files, but I have not tried out. Best, Jonas

On 18. Feb 2022, at 17:23, xavpinson @.***> wrote:

Hi,

I am trying to do some STORM images reconstruction and I am apparently unable to load tif files over 4Gb (smaller images load without any issue). I am pretty incompentent in Matlab but from what I can understand some module responsible for loading tif files seems to be the problem.

I am copying below the output I get when I try to open a tif file over 4Gb.

Any help/idea/suggestion would be very much appreciated.

Thanks!

Xavier

main directory: D:\Home\evpinto\Desktop settings directory: E:\Programmes\SMAP\application\settings plugin help directory: E:\Programmes\SMAP\application\Documentation\help D:\Home\evpinto\Desktop

err =

MException with properties:

identifier: 'MATLAB:FileIO:InvalidFid' message: 'Invalid file identifier. Use fopen to generate a valid file identifier.' cause: {} stack: [3x1 struct] Correction: [] fid =

-1 could not write file E:\Programmes\SMAP\application\Documentation\pdf\SMAP_UserGuide.pdf locally. bioformats package not found. Please select path to bioformats_package.jar in the Preferences. you can download the Matalb toolbox for bioformats athttps://www.openmicroscopy.org/bio-formats/downloads/ https://www.openmicroscopy.org/bio-formats/downloads/ Error using bfGetReader (line 51) Missing Bio-Formats library. Either add bioformats_package.jar to the static Java path or add it to the Matlab path.

Error in imageloaderOME/openi (line 18)

Error in interfaces.imageloaderSMAP/open (line 224)

Error in interfaces.imageloaderSMAP (line 37)

Error in imageloaderOME (line 14)

Error in imageloaderAll (line 42)

Error in WorkflowModules.Loaders.TifLoader/addFile (line 193)

Error in WorkflowModules.Loaders.TifLoader>loadtif_callback (line 292) simple tiff loader loader no description found could not determine number of images. Directly evaluate file, might take time. Camera not recognized. Use default Error using bfGetReader (line 51) Missing Bio-Formats library. Either add bioformats_package.jar to the static Java path or add it to the Matlab path.

Error in imageloaderOME/openi (line 18)

Error in interfaces.imageloaderSMAP/open (line 224)

Error in interfaces.imageloaderSMAP (line 37)

Error in imageloaderOME (line 14)

Error in imageloaderAll (line 42)

Error in getRoiTif (line 9)

Error in gettif (line 6)

Error in File.Load.Loader_tif>loadfile (line 82)

Error in File.Load.Loader_tif/load (line 15)

Error in File.Load.Loader_auto>loadfile (line 69)

Error in File.Load.Loader_auto/load (line 17)

Error in gui.GuiFile>loadfiles (line 333)

Error in gui.GuiFile/loadbutton_callback (line 145)

Error in gui.GuiFile>@(varargin)obj.loadbutton_callback(varargin{:}) (line 408) simple tiff loader loader could not find camera file in global settings. Using default file. Warning: Inputs must be character vectors, cell arrays of character vectors, or string arrays.

In findsettingsfile (line 9) In interfaces/imageloaderSMAP/getmetadatacam (line 179) In interfaces/imageloaderSMAP/getmetadata (line 160) In imageloaderTifSimple/openi (line 20) In interfaces/imageloaderSMAP/open (line 224) In interfaces/imageloaderSMAP (line 37) In imageloaderTifSimple (line 11) In imageloaderAll (line 47) In getRoiTif (line 9) In gettif (line 6) In File.Load.Loader_tif>loadfile (line 82) In File.Load/Loader_tif/load (line 15) In File.Load.Loader_auto>loadfile (line 69) In File.Load/Loader_auto/load (line 17) In gui.GuiFile>loadfiles (line 333) In gui/GuiFile/loadbutton_callback (line 145) In gui.GuiFile>@(varargin)obj.loadbutton_callback(varargin{:}) (line 408) In gui/GuiMainSMAP/makeGui (line 353) In SMAP (line 23)

camera calibration file settings/camera.mat not found.

err =

MException with properties:

identifier: 'MATLAB:badsubscript' message: 'Index in position 2 exceeds array bounds.' cause: {} stack: [9x1 struct] Correction: [] — Reply to this email directly, view it on GitHub https://github.com/jries/SMAP/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ52BUGU33IK47E42FU6FDU3ZW67ANCNFSM5OYMZ4XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.

Dr. Jonas Ries

Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany

Phone: +49 6221 3878199 Fax: +49 6221 387 8242 Mobile: +49 151 51040705 Email: @.*** http://www.embl.de/research/units/cbb/ries/index.html

xavpinson commented 2 years ago

Hi Jonas,

Thanks a lot for this very quick reply. We tried using the bioformat.jar but it did not seem to work. Maybe the write access problem you point out is the issue there. We'll look further into it. Thanks again for the suggestions.

Best,

Xavier

Rieslab commented 2 years ago

Dear Xavier, Let me know when you find a way of how to load the large tiff files in Matlab (even from the console), I would be very happy to integrate it in SMAP. Best, Jonas

On 22. Feb 2022, at 09:29, xavpinson @.***> wrote:

Hi Jonas,

Thanks a lot for this very quick reply. We tried using the bioformat.jar but it did not seem to work. Maybe the write access problem you point out is the issue there. We'll look further into it. Thanks again for the suggestions.

Best,

Xavier

— Reply to this email directly, view it on GitHub https://github.com/jries/SMAP/issues/5#issuecomment-1047541961, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ52BQZ7KDZUXDOOQW3X6LU4NCOJANCNFSM5OYMZ4XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.

Dr. Jonas Ries

Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany

Phone: +49 6221 3878199 Fax: +49 6221 387 8242 Mobile: +49 151 51040705 Email: @.*** http://www.embl.de/research/units/cbb/ries/index.html

cleterrier commented 2 years ago

Hi Jonas, the problem is likely that the large TIF comes from ImageJ, which doesn't really use a TIF format even if the file is saved as ".tif". Have you considered using TiffStack (https://github.com/DylanMuir/TIFFStack)? Apparently it incorporates reading the strange ImageJ format (https://github.com/DylanMuir/TIFFStack/issues/7). Anyway, sorry to hijack if that is not the actual problem of the OP.

cleterrier commented 2 years ago

Also I've open a discussion on image.sc here, as it's something that bugs me for a long time (how to read the strange ImageJ format in other software?): https://forum.image.sc/t/reading-imagej-large-tif-file-format-in-other-software/63757

Rieslab commented 2 years ago

Thanks a lot Christophe, I’ll try to integrate it once I have time and there is the need. I agree, it would be a good idea for ImageJ to save a proper compatible Tif format.

On 24. Feb 2022, at 00:14, Christophe Leterrier @.***> wrote:

Hi Jonas, the problem is likely that the large TIF comes from ImageJ, which doesn't really use a TIF format even if the file is saved as ".tif". Have you considered using TiffStack (https://github.com/DylanMuir/TIFFStack https://github.com/DylanMuir/TIFFStack)? Apparently it incorporates reading the strange ImageJ format (DylanMuir/TIFFStack#7 https://github.com/DylanMuir/TIFFStack/issues/7). Anyway, sorry to hijack if that is not the actual problem of the OP.

— Reply to this email directly, view it on GitHub https://github.com/jries/SMAP/issues/5#issuecomment-1049309098, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ52BSLELOCZG5DQV4MUD3U4VS65ANCNFSM5OYMZ4XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.

Dr. Jonas Ries

Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany

Phone: +49 6221 3878199 Fax: +49 6221 387 8242 Mobile: +49 151 51040705 Email: @.*** http://www.embl.de/research/units/cbb/ries/index.html

cleterrier commented 2 years ago

Indeed, but the issue is that ImageJ is already capable of exporting large TIFFs in proper formats like OME-TIF or BigTIFF (via BioFormats), but people don't use these options, partly because the raw "TIF-like" format is by default and also because it is faster to read/write. So there won't be much changes to hope on ImageJ's side.

Rieslab commented 2 years ago

If this is the case, I could just ask the users to export in a compatible format from ImageJ. Thanks again!

On 24. Feb 2022, at 13:44, Christophe Leterrier @.***> wrote:

Indeed, but the issue is that ImageJ is already capable of exporting large TIFFs in proper formats like OME-TIF or BigTIFF (via BioFormats), but people don't use these options, partly because the raw "TIF-like" format is by default and also because it is faster to read/write. So there won't be much changes to hope on ImageJ's side.

— Reply to this email directly, view it on GitHub https://github.com/jries/SMAP/issues/5#issuecomment-1049822723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ52BRWS3A5ZD4WH5QLA23U4YR3LANCNFSM5OYMZ4XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.

Dr. Jonas Ries

Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany

Phone: +49 6221 3878199 Fax: +49 6221 387 8242 Mobile: +49 151 51040705 Email: @.*** http://www.embl.de/research/units/cbb/ries/index.html

cleterrier commented 2 years ago

But the point is to lower the barrier to use SMAP for people coming from ImageJ. People will prefer to use something else (that understands the ImageJ format) rather than asking on a forum why SMAP doesn't read their files, then spend 15 minutes per stack to convert it to the right format. It would be much better to have something in SMAP to directly import the stacks.

xavpinson commented 2 years ago

Dear Xavier, Let me know when you find a way of how to load the large tiff files in Matlab (even from the console), I would be very happy to integrate it in SMAP. Best, Jonas … On 22. Feb 2022, at 09:29, xavpinson @.> wrote: Hi Jonas, Thanks a lot for this very quick reply. We tried using the bioformat.jar but it did not seem to work. Maybe the write access problem you point out is the issue there. We'll look further into it. Thanks again for the suggestions. Best, Xavier — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZ52BQZ7KDZUXDOOQW3X6LU4NCOJANCNFSM5OYMZ4XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented. --- Dr. Jonas Ries - group leader - Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany Phone: +49 6221 3878199 Fax: +49 6221 387 8242 Mobile: +49 151 51040705 Email: @. http://www.embl.de/research/units/cbb/ries/index.html

Hi Jonas,

I am a colleague of Xavier's and we got around the problem of not being able to upload images larger than 4GB by reducing the size of our images. Thank you for your response.

Best,

Pinto Eva