Closed ThomasMicroscopy closed 3 months ago
tischi
commented
1 year ago I @ThomasMicroscopy I had a look and it seems that MolDev files may be supported already:
/*
example:
MIP-2P-2sub_C05_s1_w146C9B2CD-0BB3-4B8A-9187-2805F4C90506.tif
well = C05, site = 1, channel = 1
*/
public static final String MOLDEV_WELL_SITE_CHANNEL = ".*_(?<"+WELL+">[A-Z]{1}[0-9]{2})_s(?<"+SITE+">.*)_w(?<"+CHANNEL+">[0-9])[^_thumb].*";Could you please try whether it works for you? Open HCS Dataset...
ThomasMicroscopy
commented
1 year ago Hmm, I seem to get error messages. The xx.HTD file is greyed out, selecting the folder with the images just gives errors during loading. Maybe the modifications for the PE format caused an issue?
(Fiji Is Just) ImageJ 2.9.0/1.53t; Java 1.8.0_322 [64-bit]; Mac OS X 10.14.6; 239MB of 35639MB (<1%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 more
Caused by: java.lang.RuntimeException: Module threw exception
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NullPointerException
at org.embl.mobie.MoBIE.initImageJAndMoBIE(MoBIE.java:250)
at org.embl.mobie.MoBIE.
tischi
commented
1 year ago Thanks for giving it a shot!
The xx.HTD file is greyed out,
What is the xx.HTD file?
Maybe the modifications for the PE format caused an issue?
What is the PE format, and how do you know they have been modified?
ThomasMicroscopy
commented
1 year ago First PE. It's PerkinElmer, who makes the Operetta and Opera HCS microscopes, which run with Harmony software. This was the format from the old thread. PerkinElmer was in the meantime bought up, and the Imaging and other divisions were spun off. So, PE is now Revvity.
Back to Molecular Devices (part of Danaher). The files have a simple structure. 1) a xxx.HTD file, basically a text file with metadata. 2) then a folder with timepoint 1 (even if there is only 1 timepoint), containing all the imaging files as TIFF , tagged with well, sites, wavelength in the file name. This folder still contains another copy of the xxx.HTD file. Some metadata are also in the TIFF files.
I don't know if the HTD file is necessary for reading the files, since the file names have all the position information.
tischi
commented
1 year ago Hi Thomas,
The title of this issue contains "Molecular Devices" (MD) thus it would be good to only discuss related things here. If you are facing PE issues, could you please open another issue dedicated to this?
Regarding the MD, could you please share a minimal dataset with me? I would like to check whether they changed their file format as compared with the system that we have here at EMBL.
ThomasMicroscopy
commented
1 year ago No, here it's only about the Molecular Devices Format. I only mentioned PerkinElmer/Harmony, because you have made modifications to the HCS importer to accommodate the PE/Harmony files, and I wondered if that had an impact on the import for other formats like Molecular Devices formats. Sorry for the confusion.
I did try to send a zipped folder with the data to you via switch file sender, but it looks like it didn't go through, when I checked again now (I let the transfer happen unattended... so much for that...). I have to send it again tomorrow.
ThomasMicroscopy
commented
1 year ago Die zipped file link should have arrived for downloading now.
best wishes Thomas
On 27 Sep 2023, at 14:31, Christian Tischer @.**@.>> wrote:
Hi Thomas,
The title of this issue contains "Molecular Devices" (MD) thus it would be good to only discuss related things here. If you are facing PE issues, could you please open another issue dedicated to this?
Regarding the MD, could you please share a minimal dataset with me? I would like to check whether they changed their file format as compared with the system that we have here at EMBL.
— Reply to this email directly, view it on GitHubhttps://github.com/mobie/mobie-viewer-fiji/issues/1056#issuecomment-1737299893, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6DLOBGFHUMROEFJWBT5ITTX4QMAJANCNFSM6AAAAAA4WJACRE. You are receiving this because you were mentioned.Message ID: @.***>
tischi
commented
11 months ago @ThomasMicroscopy I had a look and this looks doable. I will work on it.
Could you please help me with something?
In our MD microscope the filenames are like this:
MIP-2P-2sub_C05_s1_w146C9B2CD-0BB3-4B8A-9187-2805F4C90506.tif
Have you seen those before?
Do you know whether this is some different MD software version?
For you they now look like this:
Plate10x4sites_A01_s1_w1.TIF
tischi
commented
11 months ago @ThomasMicroscopy
In fact, just using the latest MoBIE release in Fiji it works out of the box for me. I don't know why you got an error...
Zooming out may crash it at some point, but just using the table to navigate from well to well works fine for me.
ThomasMicroscopy
commented
11 months ago @ThomasMicroscopy I had a look and this looks doable. I will work on it. Could you please help me with something? In our MD microscope the filenames are like this:
MIP-2P-2sub_C05_s1_w146C9B2CD-0BB3-4B8A-9187-2805F4C90506.tifHave you seen those before? Do you know whether this is some different MD software version? For you they now look like this:Plate10x4sites_A01_s1_w1.TIF
I think the files names you show, are the file names of the images directly deposited in a folder associated with the SQL database (MX uses SQL database, the files are linked to the database ).
The file names above are obtained by using the export function in MetaXpress.
Actually, we got now also the export function for InCarta activated, the XDCE file format. If that could be read as well, would be nice. I can try to make some samples.
ThomasMicroscopy
commented
11 months ago @ThomasMicroscopy
In fact, just using the latest MoBIE release in Fiji it works out of the box for me. I don't know why you got an error...
Zooming out may crash it at some point, but just using the table to navigate from well to well works fine for me.
Ok, I will have to try this again then, thank you.
ThomasMicroscopy
commented
5 months ago Hallo again, sorry for the long silence. I have tried now (with the latest Fiji and Mobie plugin - not beta) to read Molecular Devices MetaXpress file format files (.htd). OS: MS Windows 10 First loading a folder from one plate with ~ 3000 image files (96 wells, 12 FOV, 3 channels) seemed to stall. Checking with the TaskManager it maxed out the RAM (16 Gb) and probably got stuck somehow. I tried then to load a much smaller sample (~ 200 image files, 24 wells, 4 FOVs, 2 channels). It seems to load fine, but crashed then.
Here the different reports a) Log
Opening: F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262 Looking for image files... Found 192 image files in 32 ms. HCS pattern: MolecularDevices Image data format: bioformats Parsing metadata... Fetching metadata for setup 1 from F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_B04_s1_w1.TIF Fetching metadata for setup 2 from F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_B04_s1_w2.TIF Initialised HCS plate: TBO-Test31-01-2024-10x-TL-DAPI_Plate_262 Wells: 24 Sites per well: 4 Channels: 2 Frames: 1 Slices: 1
b) Console Error log
Keymap list file C:\Users\labuser\AppData\Roaming\fiji\bigdataviewer\config\keymaps\keymaps.yaml not found. Using builtin styles.
Appearance settings file C:\Users\labuser\AppData\Roaming\fiji\bigdataviewer\config/appearance.yaml not found. Using defaults.
[ERROR] Command errored: Open HCS Dataset...
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at org.embl.mobie.lib.ThreadHelper.waitUntilFinished(ThreadHelper.java:112)
at org.embl.mobie.MoBIE.initDataSources(MoBIE.java:645)
at org.embl.mobie.lib.view.ViewManager.initData(ViewManager.java:321)
at org.embl.mobie.lib.view.ViewManager.show(ViewManager.java:221)
at org.embl.mobie.MoBIE.initUIandShowView(MoBIE.java:280)
at org.embl.mobie.MoBIE.openHCSDataset(MoBIE.java:265)
at org.embl.mobie.MoBIE.
Log in Fiji/MOBIE
(Fiji Is Just) ImageJ 2.14.0/1.54f; Java 1.8.0_322 [64-bit]; Windows 10 10.0; 618MB of 11966MB (5%)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:308)
at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
at ij.IJ.runPlugIn(IJ.java)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:304)
... 5 moreCaused by: java.lang.RuntimeException: Module threw exception
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:128)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:64)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 moreCaused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at org.embl.mobie.lib.ThreadHelper.waitUntilFinished(ThreadHelper.java:112)
at org.embl.mobie.MoBIE.initDataSources(MoBIE.java:645)
at org.embl.mobie.lib.view.ViewManager.initData(ViewManager.java:321)
at org.embl.mobie.lib.view.ViewManager.show(ViewManager.java:221)
at org.embl.mobie.MoBIE.initUIandShowView(MoBIE.java:280)
at org.embl.mobie.MoBIE.openHCSDataset(MoBIE.java:265)
at org.embl.mobie.MoBIE.<init>(MoBIE.java:153)
at org.embl.mobie.command.open.OpenHCSDatasetCommand.run(OpenHCSDatasetCommand.java:87)
at org.scijava.command.CommandModule.run(CommandModule.java:196)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:125)
... 6 moreCaused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.embl.mobie.lib.ThreadHelper.waitUntilFinished(ThreadHelper.java:104)
... 16 moreCaused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at org.embl.mobie.DataStore.fetchSpimData(DataStore.java:115)
at org.embl.mobie.lib.image.ImageDataImage.openSpimData(ImageDataImage.java:185)
at org.embl.mobie.lib.image.ImageDataImage.open(ImageDataImage.java:158)
at org.embl.mobie.lib.image.ImageDataImage.getSourcePair(ImageDataImage.java:99)
at org.embl.mobie.MoBIE.initDataSource(MoBIE.java:662)
at org.embl.mobie.MoBIE.lambda$initDataSources$0(MoBIE.java:640)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
... 4 moreCaused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.embl.mobie.DataStore.fetchSpimData(DataStore.java:107)
... 10 moreCaused by: java.lang.RuntimeException: Could not open F:\USERS\XXXLab\TBO\HDT\TBO-Test31-01-2024-10x-TL-DAPI_Plate_262\TimePoint_1\TBO-Test31-01-2024-10x-TL-DAPI_C04_s3_w1.TIF
at org.embl.mobie.io.util.IOHelper.openWithBioFormatsFromFile(IOHelper.java:166)
at org.embl.mobie.io.util.IOHelper.openWithBioFormats(IOHelper.java:144)
at org.embl.mobie.lib.hcs.VirtualBioFormatsStack.getProcessor(VirtualBioFormatsStack.java:213)
at ij.ImagePlus.setStack(ImagePlus.java:843)
at ij.ImagePlus.<init>(ImagePlus.java:162)
at org.embl.mobie.DataStore.openSpimData(DataStore.java:237)
at org.embl.mobie.DataStore.lambda$null$2(DataStore.java:106)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)Do you have any idea what the problem could be? Best wishes TB
tischi
commented
5 months ago @ThomasMicroscopy could you please share some of those files with me? Maybe the first two wells?
ThomasMicroscopy
commented
4 months ago sorry for the delay, I'll send a zip file via switch.
tischi
commented
4 months ago Working on this here: issue-1056
tischi
commented
4 months ago Dear @ThomasMicroscopy,
I downloaded your data and tested it.
It appears that some of the images cannot be opened with Bio-Formats using Plugins › Bio-Formats › Bio-Formats Importer.
For example, this file TBO-Test31-01-2024-10x-TL-DAPI_Plate_260/TimePoint_1/TBO-Test31-01-2024-10x-TL-DAPI_E04_s6_w2.TIF throws this error:
java.lang.ArrayIndexOutOfBoundsException: -1
at loci.formats.in.CellWorxReader.getFile(CellWorxReader.java:715)
at loci.formats.in.CellWorxReader.populateMetadata(CellWorxReader.java:438)
at loci.formats.in.CellWorxReader.initFile(CellWorxReader.java:310)
at loci.formats.FormatReader.setId(FormatReader.java:1466)
at loci.plugins.in.ImportProcess.initializeStack(ImportProcess.java:518)
at loci.plugins.in.ImportProcess.execute(ImportProcess.java:145)
at loci.plugins.in.Importer.showDialogs(Importer.java:156)
at loci.plugins.in.Importer.run(Importer.java:77)
at loci.plugins.LociImporter.run(LociImporter.java:78)
at ij.IJ.runUserPlugIn(IJ.java:244)
at ij.IJ.runPlugIn(IJ.java:210)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)To further investigate and eventually fix this we would need to create a post on forum.image.sc, providing the image file as an example for download. Could you please create such a post and ping me there with @Christian_Tischer?
ThomasMicroscopy
commented
4 months ago Ok, I'll open a thread there. I noticed "arrayindexoutofbounds" - maybe the newer camera resolutions cause a problem. This image is 2048 x 2048. Or the binning. I can try binning 2x, gives 1024x1024.
ThomasMicroscopy
commented
4 months ago it's now on https://forum.image.sc/t/opening-metaxpress-6-7-2-data-htd-in-bio-formats/96860, not sure if that "ping" worked, never used it.
tischi
commented
4 months ago Thanks, I answered there.
Note to self:
tischi
commented
4 months ago Hi @ThomasMicroscopy,
Maybe that’s the issue with mobie, you don’t use the xx.htd, but the tif file, and then it fails.
This is correct, but I cannot easily support opening such "meta-files", such as the xx.htd, in MoBIE due to the internal architecture of the code in MoBIE. I can only support opening individual files and arrange them into a plate format based on their naming scheme. Thus, we must find out in the forum post (with the help of the ImageJ and Bio-Formats developers) how to open the individual files.
And again, sorry for the misunderstanding, the forum post should not be about MoBIE but just about how to open these images in Fiji. Everything MoBIE related we can discuss here.
Would you mind testing whether you could convert your HCS data to OME-Zarr? https://forum.image.sc/t/ngff-converter-2-0-released-convert-bioformats-to-ome-ngff-ome-tiff/96102/3
tischi
commented
4 months ago ...and are you planning on uploading one of the files to the forum? I can also do it for you if you give me the permission (I didn't want to publicly share your data just like that).
ThomasMicroscopy
commented
4 months ago Yes, you have the permission to upload a file. As far as I can see behave all tiff files in the folder this way. The only way to open with Bio-Formats is clicking on the xx.HTD file. And MoBIE can open the TIFF files also, if one chooses "Open Image..." and select an individual file. I think the plate arrangement is in the xx.HTD file, and without that information it doesn't work.
Thank you very much for the help.
ThomasMicroscopy
commented
4 months ago I added some new information on the image forum. It seems the crash problem only happens after installation of MoBIE.
tischi
commented
3 months ago Thanks! And I updated MoBIE, could you please test whether it works for you now?
Hello support,
We have received a new Molecular Devices instrument. I was wondering, if the HTD format support could be provided in Mobile-Viewer-Fiji. I could generate files now.
best wishes