Slow auto-contrast?!

[tibuch]

Hi,

I am missing the time-slider in the newest version of MoBIE. I downgraded my MoBIE again and the same dataset has the time-slider. Screenshot from the old MoBIE with the time-slider at the bottom.

This is a single-channel 3D + Time dataset.

I tried the deprecated and the newest way of opening a single ome-zarr.

[tischi]

I cannot reproduce it. I created an OME-Zarr in Fiji and opened it:

  new ImageJ().ui().showUI();
  final OpenImageAndLabelsCommand command = new OpenImageAndLabelsCommand();
  command.image = new File( "/Users/tischer/Desktop/tim-oliver.ome.zarr" );
  command.run();

[tibuch]

Oh, it works! But it is very slow (like 5min slow). I didn't wait long enough before and thought that nothing is happening.

I just tried again and got distracted and suddenly MoBIE opens and displays the data.

I like that the display range is set to reasonable numbers, but could this take so long?

Once MoBIE is open I can easily scroll through the data. Minimal (<1sec) load times if I access a chunk which is not cached.

[tischi]

I added some auto-contrast functionality, but depending on the how the data is stored this indeed seems to be very slow. I probably need to disable it again (at least for some data). What exactly did you load such that it was slow? Was it an HCS plate?

[tibuch]

Only a single image: .zarray:

{
    "chunks": [
        1,
        1,
        512,
        512
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 3,
        "cname": "zstd",
        "id": "blosc",
        "shuffle": 1
    },
    "dimension_separator": "/",
    "dtype": "<u2",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
        224,
        1,
        512,
        512
    ],
    "zarr_format": 2
}

I don't know what your heuristic is, but maybe it would be sufficient to compute them only on the field of view which is displayed?

[tischi]

That means you don't have a resolution pyramid here? That would be the issue then...and yes I definitely need to restrict it to the currently visible data, that unf. is just a bit of work....but I will need to do it.

[tibuch]

Yes, there is no resolution pyramid. For this data the YX size is small enough, but time can be quite long.

[tischi]

I will release a version later today where I will disable all the auto-contrast again and then revisit this when I have a bit more time.

[tischi]

@tibuch I pushed a new version to the master branch with this modification.

Would you have time to build it and test from source and see whether that works, e.g. putting a breakpoint there and see what it does and whether it is faster now? It would be very helpful if we could debug this together....

Also I'd be very curious if the lines above are fast where I determine the number of time points and z slices, because they also already "touch" the data....

[tischi]

Hi @tibuch, the data you shared with me opens very fast. However, I am puzzled because I don't see how my recent changes should have improved this, i.e. it should have been fast before....maybe you had local network issues?

Anyway, I uploaded a new version, please test it.

[tibuch]

Should this open the AmazonS3ClientBuilder?

It is still slow for me, but I think it got a bit faster.

The drive is mounted via samba. Which is known to be "slowish", but once the data is open loading chunks is fast and before I never had this opening issue.

[tibuch]

It is super fast if I open a local copy.

[tischi]

Should this open the AmazonS3ClientBuilder?

I always initialise the AmazonS3ClientBuilder, just in case, because this can be slow, but this is running in a different thread so I would hope that it should not affect the loading of your data...but, who knows...ah, but since you are saying now that a local copy is super fast, then maybe it really is just your samba mount?

[tibuch]

Looks like it is related to the samba mount. I will try to find some time this afternoon to debug this.

Is there any directory tree parsing happening?

[tischi]

Is there any directory tree parsing happening?

Yes, probably the AmazonS3ClientBuilder is looking all over the place to find potential places where Amazon S3 Credentials could be. I already made the experience that this takes very different amounts of time on different computers.

1. How long does this take for you? It should print it.
2. If you have the time to run from a build from source and comment out these lines of code and see whether this speeds up things that would be very interesting!
3. Also please try whether it makes a difference if the samba mount is present on your computer, even if you load the data from local! That would tell us whether the AmazonS3ClientBuilder is the issue.

[tischi]

And, the Janelia N5 library that I am using now to open OME-Zarr probably does quite a bit more directory parsing to open the OME-Zarr than our old code did....

[tibuch]

3. Also please try whether it makes a difference if the samba mount is present on your computer, even if you load the data from local! That would tell us whether the AmazonS3ClientBuilder is the issue.

The samba mount was present during this test this morning. The only difference was the path pointing to a local directory.

[tibuch]

1. How long does this take for you? It should print it.

This can't be it: ( Initialised AmazonS3ClientBuilder in 963 ms. )

[tischi]

One theory: the N5 library prefetches quite some data from the OME-Zarr; it does so highly parallel, which makes it rather efficient when interacting with object stores; it may be that certain storage systems (like your samba mount and the underlying hardware) may not play well with highly parallel requests. If this turns out to be the issue we would need to get in touch with John Bogovic to see whether they tested opening OME-Zarr in such scenarios.

[tischi]

To test to above hypothesis in a cleaner way I recommend benchmarking in Fiji:

File › Import › HDF5/N5/Zarr/OME-NGFF ...

or

Plugins › BigDataViewer › HDF5/N5/Zarr/OME-NGFF Viewer

[tischi]

@tibuch please also have a look here: https://forum.image.sc/t/mobie-slow-when-saving-data/98730/5

It could be that there is something with the new N5 library that does not play well with remote file stores...

[tibuch]

Plugins › BigDataViewer › HDF5/N5/Zarr/OME-NGFF Viewer

This works instantly.

[tischi]

OK, if the above works instantly, then we have to find out which part of my code is taking all this time. Since I don't have your setup it would really be great if you could build MoBIE from source and set a few breakpoints (or use VisualVM) to see what takes so long...

[tibuch]

Current main branch of mobie-viewer-fiji should be enough?

[tischi]

Yes, main branch of mobie-viewer-fiji would be it....

But, to help debugging, I also just now added these examples to the main branch of mobie-io:

• OpenOMEZarrExample
• N5ViewerExample

Before looking into mobie-viewer-fiji, it would be important to test within mobie-io whether

1. The N5ViewerExample has the same performance as Plugins › BigDataViewer › HDF5/N5/Zarr/OME-NGFF Viewer
2. The OpenOMEZarrExample has the same performance as the N5ViewerExample

If any of those differ we have something to work with. If they are all the same then I think we have to look into additional sources of slowness in mobie-viewer-fiji.

---

Within mobie-viewer-fiji this would be your entry point:

• https://github.com/mobie/mobie-viewer-fiji/blob/main/src/test/java/debug/DebugIssue1167.java

I was so far mainly looking into the setMetadata function of ImageGridSources in terms of finding something that could be slow; and I would also recommend this for you as a starting point...

[tibuch]

Good morning!

I found the issue, it is here: https://github.com/mobie/mobie-viewer-fiji/blob/59d0646a48e31e068c8410f545dda2ddba4701c4/src/main/java/org/embl/mobie/lib/MoBIEHelper.java#L338

It looks like this takes a long time once we reach the multiscale directory structure. There it parses each Timepoint, Z, and Y directory. Would it be possible to stop going deeper once a .zarray is encountered?

If I set maxDepth = 2 it works very fast. But I guess this is not a viable option.

[tibuch]

Maybe this requires a custom FileVisitor which uses SKIP_SUBTREE to avoid all the chunk-directories.

[tischi]

Great catch! I will also have a look and think about it...maybe not today, but at some point this week.

[tischi]

I just set the depth to 2 now (uploaded to Fiji). Honestly, I don't really know anymore for which use-cases I needed a depth > 1 at all; let's just see if that breaks anything 😅 ...

[tibuch]

:sweat_smile: Maybe something with detecting labelings (or other sub-groups) inside plates? I think depth 2 could lead to issues in a plate-layout, where the multiscale images start on level 5 or so.

[tischi]

I checked, I am using other code for that...

[tischi]

Can we close this?