Open ThomasMicroscopy opened 1 year ago
Hi, Yes, this should be possible, but I won't have time this week to look at it. Please feel free to ping me again at some point in case I did not get back to you.
@ThomasMicroscopy I just downloaded the data. I tried to open an individual TIFF file in Fiji but it threw an error. Is that the same for you? Opening via the XML does work.
Yes, opening a single .TIFF file doesn't work. One specifically has to drag the xml onto Fiji, then you get the dialog to choose which images from the plate to open. This is how PE supports Bioformats, according to my understanding.
I hate this... ;-) I mean how should one work with such data...
Indeed, what's the point of a tiff that one can't open... Actually, it's not consistent, I have some small test images from the CLS. For example, just tried now a sample which consists only of one field in one well with three channels, i.e. 3 tiffs. These I can open with Fiji as single files with no issue. But another folder, with 48 images (1 well, a couple of fields, several channels) doesn't open after the import dialog, generates errors.
As to how to work with such data? Well, use the Harmony software... ...
Hello, any progress with this? best wishes
Not yet, sorry!
no problem. I just saw some Fiji updates for mobie, so was wondering (but still crashed with the 5.1 files)
best wishes Thomas
On 12 Apr 2023, at 15:47, Christian Tischer @.**@.>> wrote:
Not yet, sorry!
— Reply to this email directly, view it on GitHubhttps://github.com/mobie/mobie-viewer-fiji/issues/1007#issuecomment-1505310400, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6DLOBFVJGQMD3CE4RMPJWLXA2XANANCNFSM6AAAAAAWBFDJMQ. You are receiving this because you were mentioned.Message ID: @.***>
@ThomasMicroscopy I am having another go at this and this will require some substantial work as it is not possible to open those images individually in Fiji. I will see how far I get....
thank you. If it doesn’t work easily, you could put it low down in the pipeline, since we still use Harmony 4.1.
Anyway, how can one make a TIFF format so that it cannot be opened…. what are they thinking.
Thanks for your help.
best wishes Thomas
edit: Removed MolDev content not relevant to this thread...
I will work on it now.
Let's please in this issue only discuss Operetta 5.
If you would like to talk about MolDev, please open a new issue with an appropriate title :-)
@ThomasMicroscopy
please see me post above
and could it be that the sample images that you sent me look like this?
Oh, it could also be that we did not yet implement code for having 49 sites/fields... Let me check.
Ok, I think i fixed it:
yes, way better ;-)
Here is the overview of the first well, composite of all 3 channels. Your image is only channel 2 (red), which is more sparse then channel 1 (blue nuclei), or channel 3 (gfp)
As to number of fields/sites... Scanning a 96 plate well with 40 x, gives even more fields.....
Ok, think it is working. I am not sure yet, but for some reason it takes a while for the images to load completely:
It first looks like this:
and only after some time like this:
Yes, this looks right, except the color choice for the channels is different.
As to slow loading, it's about 86 GB of data, so loading everything depends a lot on the hardware, i.e. RAM, local hard drive speed (HDD or SSD ?), connection (SATA, M.2/PCIe...) , external drive (HDD, SSD ?, connection speed), etc...
The Harmony 4.1 data take quite a while to display as well on my system. Sometimes I have the feeling it could be faster, if the system just forces sequential reads of the images, rather than doing a lazy approach (I think that's what your are calling it?). I guess that's why some systems produce also thumbnails, for quick overviews.
To make the Harmony 5 work I now need to directly load the XML instead of the files.
For this I need to use some different image loader library and I don't have full control here what is happening.
To reduce my maintenance work I now also for Harmony 4 load the images via the XML.
I uploaded now the changes to MoBIE-beta.
Please test it and let me know.
Regarding the colours and contrast settings: as I cannot read them anymore from the image files, maybe I could read them from the XML? Do you know?
As to colors, yes, there is filter information in the xml for the channels. So far I just set it by hand.
As to loading, it seems to have an issue, even the Harmony 4.1 files fail:
(Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.14.6; 244MB of 26843MB (<1%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw error
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 more
Caused by: java.lang.RuntimeException: Module threw error
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:130)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NoClassDefFoundError: ch/epfl/biop/bdv/img/CacheControlOverride
at org.embl.mobie.io.SpimDataOpener.setSharedQueue(SpimDataOpener.java:167)
at org.embl.mobie.io.SpimDataOpener.openWithBioFormats(SpimDataOpener.java:302)
at org.embl.mobie.lib.hcs.Plate.
Since I see an EPLF problem, I am wondering if your plugin has a conflict with some epfl plug-in (BIOP plug-in, PTBIOP update site). I think there was also an update from epfl when I installed the MoBIE updates now.
I removed the BIOP plug-in from the PTBIOP site, but still the same problem it looks like. (as an aside, the BIOP plug-in doesn't seem to work anymore either).
(Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.14.6; 485MB of 26843MB (1%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw error
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 more
Caused by: java.lang.RuntimeException: Module threw error
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:130)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NoClassDefFoundError: ch/epfl/biop/bdv/img/CacheControlOverride
at org.embl.mobie.io.SpimDataOpener.setSharedQueue(SpimDataOpener.java:167)
at org.embl.mobie.io.SpimDataOpener.openWithBioFormats(SpimDataOpener.java:302)
at org.embl.mobie.lib.hcs.Plate.
Yes, thanks! Some dependencies changed... Does it work now?
hmm, not sure if I should continue with this.... (when I try the updater)
Hmm, strange, maybe easiest is try with a fresh Fiji?
ok, I inactivated the PTBIOP, now this error is gone.
it seems both MoBIE and PTBIOP use this bigdataviewer... and have maybe different versions ??
ok, it's getting further, this is Harmony 5.1, the data I sent you. still seems to hang though (data too big form my iMac?)
Harmony 4.1 data also seem to have the same problem, looks the same
(Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.14.6; 378MB of 26843MB (1%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 more
Caused by: java.lang.RuntimeException: Module threw exception
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at org.embl.mobie.lib.ThreadHelper.waitUntilFinished(ThreadHelper.java:110)
at org.embl.mobie.MoBIE.initDataSources(MoBIE.java:1030)
at org.embl.mobie.lib.view.ViewManager.initData(ViewManager.java:348)
at org.embl.mobie.lib.view.ViewManager.show(ViewManager.java:251)
at org.embl.mobie.MoBIE.initUIandShowView(MoBIE.java:489)
at org.embl.mobie.MoBIE.openHCSDataset(MoBIE.java:474)
at org.embl.mobie.MoBIE.
No, this seem to be some other error.
Is this with the example data that you sent me? It would be good to first try with this.
yes, that's what I sent you. (well, actually, there would be two recordings to the same plate).
OK, I will try again and upload again (I will let you know).
It is this data, correct?
Just to be sure, how many files does the folder have for you?
10585 files, file names, time stamps look the same.
Just tried to open this with a fresh install of Fiji, and new downloads from mobile-beta
PS: I'm (still) using OSX Mojave (10.14) - (32 bit/ 64 bit ???)
still the same error
(Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.14.6; 1650MB of 38350MB (4%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 more
Caused by: java.lang.RuntimeException: Module threw exception
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at org.embl.mobie.lib.ThreadHelper.waitUntilFinished(ThreadHelper.java:110)
at org.embl.mobie.MoBIE.initDataSources(MoBIE.java:1030)
at org.embl.mobie.lib.view.ViewManager.initData(ViewManager.java:348)
at org.embl.mobie.lib.view.ViewManager.show(ViewManager.java:251)
at org.embl.mobie.MoBIE.initUIandShowView(MoBIE.java:489)
at org.embl.mobie.MoBIE.openHCSDataset(MoBIE.java:474)
at org.embl.mobie.MoBIE.
just noticed there is a slightly newer version of Fiji, but after updating, still the same problem
Ah, I have much less files:
Opening: /Users/tischer/Desktop/CLSsamples
HCS Pattern: Operetta
Parsing XML: /Users/tischer/Desktop/CLSsamples/Index.xml ...
Images in XML: 10584
Files: 150. // <----- HERE
Initialised HCS plate: CLSsamples
Images: 147
Channels: 3
Wells: 1
Sites per well: 49
HCS Pattern: Operetta
Maybe we are not looking at the same data?!
This is at the beginning of the XML:
<PlateID>CO001_FOXO1_DsRED_FullPLate_x20Air</PlateID>
<MeasurementID>34aadcb6-36fe-480d-af8f-db49c2bc94ae</MeasurementID>
<MeasurementStartTime>2022-06-15T11:48:56.2280644+02:00</MeasurementStartTime>
Are you using the same one?
yes, this is the same data set, but I forgot, I sent you only a limited set of images as a start, not all the wells. I forget, maybe I never sent you the full set (which I am trying to load now, but fail)
Can you try just loading the first 150 files?
The last one has this name: r01c01f49p01-ch3sk1fk1fl1.tiff
ok, I made a folder, with just the images for the first well. It still doesn't look right
here an excerpt of the full text of the log:
Error in opener Unhandled pixel type for serie 0: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":0,"nChannels":-1} Error in opener Unhandled pixel type for serie 1: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":1,"nChannels":-1} Error in opener Unhandled pixel type for serie 2: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":2,"nChannels":-1} Error in opener Unhandled pixel type for serie 3: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":3,"nChannels":-1} Error in opener Unhandled pixel type for serie 4: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":4,"nChannels":-1} Error in
In the log is just this error repeated over and over: Error in opener Unhandled pixel type for serie 4: int8 : {"opt":"","unit":"MICROMETER","nReader":10,"defaultBlockSize":true,"blockSize":[512,512,1],"splitRGB":false,"type":"BIOFORMATS","location":"/Volumes/DATA/copied first images foxo CLS/Index.xml","id":4,"nChannels":-1}
The xml should be the same as you have (in theory). I'll email it.
Opening works for my by dragging the xml on FIJI. both, the full folder, and the first well only folder.
I checked, all fiji stuff I can find on my computer is 64 bit. I'm still running Mojave (10.14) to have compatibility with some 32 bit apps - but seems none of the fiji stuff is amongst this, though I have some java apps.
Actually, I just got to open a Harmony 4.1 dataset, so that seems to work now (again).
However, I did find an out of sync problem with this Harmony 4.1 dataset. There is a missing field (error during recording), and it looks like after that missing field, the fields in all following wells are shifted, giving a wrong assembly. First, full overview, then a zoomed in section where the problem starts.
This is getting to complex, could we please continue the Harmony 4 discussion in the other issue that I just created.
Hm, really weird that I don't get this error; but also strange that you only get this for the Harmony 5...
What is also weird that it complains about int8
, because your images are int16
.. . When you open them directly in Fiji what bit-depth do they have?
I also don't understand why your images are so far apart from each other in the screenshot. What do you use as an "Relative Site Margin" ? I am using 0.0
...
relative site margin is default, 0.0, and relative well margin 0.1
Note to self, this is the branch/PR: https://github.com/mobie/mobie-viewer-fiji/pull/1018
@ThomasMicroscopy
Please see this thread: https://forum.image.sc/t/loading-hcs-plates-via-index-xml-with-bio-formats/79932/6
Apparently @NicoKiaru already wrote some code to open Operetta HCS files. Could you please test whether this works for you?
Ah, I figured how how this works. I was able to find the command now. Alas, it doesn't work with Harmony 5.1 : I get this error: Error, file /Volumes/Extreme SSD/Foxo/CLSFoxoOriginalsCLS/CO001_FOXO1_DsRED_FullPLate_x20Air__2022-06-15T11_48_56-Measurement 1/Images/Index.idx.xml not found! I have not tried a simple rename to index.xml yet.
As to Harmony 4.1 data, it does read, but generates errors, and seems to display only the first field.
lots of warnings (probably ok), but then error:
..... WARN] Unknown AcquisitionMode value 'NonConfocal' will be stored as "Other" Spimdata mpicbg.spim.data.SpimData@45771ab1 is using the Bdv Playground global cache. [ERROR] Unmatched Other Sources [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Angle [ERROR] Unmatched Channel [ERROR] Unmatched Displaysettings [ERROR] Unmatched FileIndex [ERROR] Unmatched Illumination [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Angle [ERROR] Unmatched Channel [ERROR] Unmatched Displaysettings Exception in thread "AWT-EventQueue-0" [ERROR] Unmatched FileIndex java.lang.ArrayIndexOutOfBoundsException: 9 >= 9 at java.util.Vector.elementAt(Vector.java:479) at javax.swing.tree.DefaultMutableTreeNode.getChildAt(DefaultMutableTreeNode.java:245) at javax.swing.tree.DefaultTreeModel.nodesWereInserted(DefaultTreeModel.java:313) at sc.fiji.bdvpg.scijava.services.ui.SourceFilterNode.lambda$insert$1(SourceFilterNode.java:163) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82) [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Angle [ERROR] Unmatched Channel [ERROR] Unmatched Displaysettings [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Angle [ERROR] Unmatched Channel [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Channel [ERROR] Unmatched Tile [ERROR] Unmatched Other Sources [ERROR] Unmatched All Sources [ERROR] Unmatched Channel [ERROR] Unmatched Tile [WARNING] 2 exceptions occurred during plugin discovery. Reading IFDs
@ThomasMicroscopy did you export the dataset or try to open the archive ? I do not get why you have a html file and not an xml file...
Can you share the xml file ? (no need for tiffs)
just a typo, no html. but based on the error it's looking for "Index.idx.xml", but the Harmony 5.1 export index files are called "index.xml".
can you get this:
https://drive.switch.ch/index.php/s/Z28pRfRd9Pi4jK6
and this for a harmony 4.1 export file
PS: the "operetta importer ..." from PTBIOP works. (latest update from today, there was an error).
Hello, Now that the Harmony 4.1 HCS importer is working (thank you very much), I was wondering whether it is also possible to get the HCS importer to work for the newer devices (e.g. Operetta CLS) that use Harmony 5.1? I have sent a sample file set via the file sender.
best wishes Thomas