Add OME-Zarr HCS plate support

[tischi]

@tibuch

1. Are you using MoBIE to view the official OME-Zarr plate spec?
2. Are you using the official OME-Zarr plate spec at all?

[tibuch]

1. Yes
2. Yes

I am working/finishing on an example notebook which just writes the necessary MoBIE views for a ome-zarr plate.

[tischi]

Great! Once you have it, please share such a dataset.json with me. I would then also add support to directly open such a plate (i.e. the Java code would create the necessary views on the fly).

[JensWendt]

As I am trying out stuff with S3 and OME-zarr at the moment, may I ask if there was any progress regarding opening an OME-zarr plate from S3?

[tischi]

@JensWendt thank you for your interest. No progress yet. But I think spending one day (or less) on it may be enough to fix it.

Issue is that currently I am using this code to find the paths to the sites:

        if ( hcsDirectory.endsWith( ".zarr" ) )
        {
            hcsPattern = HCSPattern.OMEZarr;
            imageDataFormat = ImageDataFormat.OmeZarr;

            final int minDepth = 3;
            final int maxDepth = 3;
            final Path rootPath = Paths.get(hcsDirectory);
            final int rootPathDepth = rootPath.getNameCount();
            imageSitePaths = Files.walk( rootPath, maxDepth )
                    .filter( e -> e.toFile().isDirectory() )
                    .filter( e -> e.getNameCount() - rootPathDepth >= minDepth )
                    .map( e -> e.toString() )
                    .collect( Collectors.toList() );
        }

The code is here.

This is fine on a file system but too slow on S3. On S3 one would need to replace this code by fetching this information by parsing the top level .zattrs JSON file; this would be better anyway also for the file system.

1. Are you able to write Java code? You'd be warmly welcome to contribute to this project. Or maybe @tibuch would like to give it a shot?
2. Do you have a use case for this? If you really need this it raises the probability that I will work on it sooner ;-)

Forum post

[tischi]

Here is one HCS example data set on S3: https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/5966.zarr

[tischi]

I am working on it https://github.com/mobie/mobie-viewer-fiji/pull/1064

[tischi]

Couple of hours of work, seems to be working for: https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/5966.zarr

[JensWendt]

Hello @tischi , what a ride... First of all, thank you! My expectation was to just get an answer how high of a priority this is for you folks now. To answer your questions: No, I am not familiar with Java, only Python. No, there is no concrete use case. I am playing around trying to do some proof-of-concept stuff to then bring this to our users. It looks great! Can I test around with it somehow?

[tischi]

@JensWendt I would be great if you could test it and report bugs and feature requests here.

It is available right now on MoBIE-beta.

[JensWendt]

Heyo, I threw in a plate imaged on our Opera system. 384 wells, 2 images per well, 3 channels, only 2D (x,y). I converted the original .flex files with the NGFF-Converter from Glencoe. The plate works fine on the vizarr and itk viewers. I get on the ImageJ console:

[ERROR] Command errored: Open HCS Dataset...
java.nio.file.InvalidPathException: Illegal char <:> at index 5: https://radosgw.public.os2.wwu.de/ngff/Opera.zarr/A/12/0\.zgroup
    at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
    at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
    at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
    at java.nio.file.Paths.get(Paths.java:84)
    at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.groupExists(N5OmeZarrReader.java:237)
    at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.exists(N5OmeZarrReader.java:286)
    at org.embl.mobie.io.n5.shaded.N5FSReader.<init>(N5FSReader.java:118)
    at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.<init>(N5OmeZarrReader.java:112)
    at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.<init>(N5OmeZarrReader.java:129)
    at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.<init>(N5OmeZarrReader.java:175)
    at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.readFile(OMEZarrOpener.java:75)
    at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.openFile(OMEZarrOpener.java:54)
    at org.embl.mobie.io.SpimDataOpener.openOmeZarr(SpimDataOpener.java:218)
    at org.embl.mobie.io.SpimDataOpener.open(SpimDataOpener.java:110)
    at org.embl.mobie.lib.hcs.Plate.<init>(Plate.java:104)
    at org.embl.mobie.MoBIE.openHCSDataset(MoBIE.java:266)
    at org.embl.mobie.MoBIE.<init>(MoBIE.java:156)
    at org.embl.mobie.command.open.OpenHCSDatasetCommand.run(OpenHCSDatasetCommand.java:66)
    at org.scijava.command.CommandModule.run(CommandModule.java:196)
    at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
    at org.scijava.module.ModuleRunner.call(ModuleRunner.java:125)
    at org.scijava.module.ModuleRunner.call(ModuleRunner.java:64)
    at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:247)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

and in the Log:

# MoBIE
Opening: https://radosgw.public.os2.wwu.de/ngff/Opera.zarr
Parsing 384 wells...
Initialising (1/384): A/2
Initialising (2/384): A/9
Initialising (4/384): A/13
Initialising (8/384): A/10
Initialising (16/384): A/16
Initialising (32/384): B/8
Initialising (64/384): C/16
Initialising (128/384): F/8
Initialising (256/384): K/16
Parsed 384 wells in 1278 ms, using up to 15 thread(s).

the .zgroup it references reads:

{
  "zarr_format" : 2
}

the same as other .zgroup on that level.

I have no clue what the issue is. Any ideas?

[JensWendt]

Same with another plate that I took of the official NGFF example page. Illegal char <:> at index 5: https://radosgw.public.os2.wwu.de/ngff/test.zarr/A/3/0\.zgroup

[tischi]

This should be fixed now in the latest MoBIE-beta (see https://github.com/mobie/mobie-viewer-fiji/issues/1065 for more details).

[JensWendt]

Works like a charm!! Many thanks. I will start promoting this to users

[tischi]

Nice! Please consider citing the publication: https://www.nature.com/articles/s41592-023-01776-4

[tischi]

@JensWendt this is now also available on the normal MoBIE update site. So no need to do the complicated MoBIE-beta stuff. Important: uncheck the MoBIE-beta update site when you activate the normal one; otherwise there will be conflicts!