IDR / omero-mkngff

Plugin to swap OMERO filesets with NGFF
GNU General Public License v2.0
0 stars 2 forks source link

No sql option #9

Closed will-moore closed 10 months ago

will-moore commented 11 months ago

NB: this is on top of #8.

In cases where we've run a lengthy omero mkngff sql... command on e.g. a test server (e.g. idr-testing) and we wish to reuse the generated sql files on a different clean server (starting from the same state - same Fileset IDs, paths/to/fileset.zarr etc, we still need to create the symlinks on the new server, without walking the fileset.zarr which can take a long time.

This PR adds a symlink command to ONLY create symlinks.

E.g:

$ omero mkngff symlink /OMERO/ManagedRepository 1161 /path/to/fileset.zarr
will-moore commented 11 months ago

...
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr -> /bia-integrator-data/S-BIAD867/f791de8d-cd01-4303-8b08-67cbbbb45b64/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr -> /bia-integrator-data/S-BIAD867/fcc7c91a-f0e6-43f4-93a6-220e9224eda5/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr
will-moore commented 11 months ago

Viewing image from that last Fileset above (Plate P127): http://localhost:1080/webclient/?show=image-693716 gives:

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr/.zattrs

But the data (symlink) is correct:

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls -alh /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr/
total 52K
drwxr-xr-x. 2 root root 4.0K Sep  4 20:58 .
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 ..
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 A
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 B
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 C
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 D
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 E
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 F
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 G
drwxr-xr-x. 2 root root 4.0K Sep  6 08:48 H
-rw-r--r--. 1 root root  12K Sep  4 20:58 .zattrs
-rw-r--r--. 1 root root   24 Sep  4 20:58 .zgroup

And Blitz logs have:

2023-09-15 07:59:33,662 ERROR [                ome.io.nio.PixelsService] (l.Server-2) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr/.zattrs
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
    at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
...
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
    at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
    at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
    at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
    at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)
    at loci.formats.FormatReader.setId(FormatReader.java:1443)
    at loci.formats.ImageReader.setId(ImageReader.java:849)
    at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
will-moore commented 11 months ago

TODO: test whether there's an issue with creating symlinks and generating sql separately... (do both, independently on the same server), then run sql and...?