ZarrReader performance improvements

ome / ZarrReader

Other

5 stars 8 forks source link

ZarrReader performance improvements #64

Closed dgault closed 9 months ago

dgault commented 1 year ago

Initial performance improvements for ZarrReader related to https://github.com/ome/ZarrReader/issues/63

This PR is an initial PR for testing purposes, in its current state it looks to reduce the initFile time by about 50%, hopefully further improvements can be made.

Based on a subset of idr0036 the initial profiling looked as below,

The updated profiling should look more like this (the Zarrutils.toJSON should also be gone). If we can manage to remove some of the remaining JZarr calls then this may be reduced further:

The initial fixes here target the below areas:

Reduce the unnecessary parsing and storage of attribute files in XMLAnnotations. This should also reduce the size of the memo file. An option has been added to allow this to be configurable if needs be.
Reduce the number of calls JZarr to open the .zarray. These calls were taking a large portion of the initialization time, mostly occurring on setSeries. I have tried to reduce this down but it is atill required for retrieving the shape of each array to populate the dimension sizes.
Reduce the number of calls JZarr to fetch group attributes. These calls were slow and as such I have tried to reduce the level of duplication. Hopefully I can get this down further still.

snoopycrimecop commented 1 year ago

Conflicting PR. Removed from build BIOFORMATS-push#94. See the console output for more details. Possible conflicts:

PR #53 dgault 'Reorder group keys'
- src/loci/formats/in/ZarrReader.java
PR #58 dgault 'Add new option to list pixels in getUsedFiles'
- src/loci/formats/in/ZarrReader.java

--conflicts

snoopycrimecop commented 1 year ago

Conflicting PR. Removed from build BIOFORMATS-push#22. See the console output for more details. Possible conflicts:

PR #53 dgault 'Reorder group keys'
- src/loci/formats/in/ZarrReader.java
PR #58 dgault 'Add new option to list pixels in getUsedFiles'
- src/loci/formats/in/ZarrReader.java

~~--conflicts~~ Conflict resolved in build BIOFORMATS-push#95. See the console output for more details.

will-moore commented 1 year ago

Updated ZarrReader on idr-testing as described at https://github.com/IDR/idr-metadata/issues/672#issuecomment-1713858312 (after having used mkngff on 3 Plates from idr0025). Then tried viewing images from Plate 2 for the first time. Logs say it took 95 mins for setid, but images appear black - see https://github.com/IDR/idr-metadata/issues/672#issuecomment-1714112834

will-moore commented 1 year ago

Today's build took 4 mins for setId on idr0025 Plate 3: https://github.com/IDR/idr-metadata/issues/672#issuecomment-1715552599

snoopycrimecop commented 12 months ago

Conflicting PR. Removed from build BIOFORMATS-push#99. See the console output for more details. Possible conflicts:

PR #53 dgault 'Reorder group keys'
- src/loci/formats/in/ZarrReader.java

~~--conflicts~~ Conflict resolved in build BIOFORMATS-push#100. See the console output for more details.

will-moore commented 12 months ago

Testing "black" images OMEZarrReader using build from 11th Sept when I first noticed it (today's didn't include this PR)...

https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/534/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Testing on idr0125-pilot...

ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1040:localhost:80
sudo -u omero-server -s
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/534/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/

# as wmoore...
sudo service omero-server restart

Tried viewing a Plate P170 from idr0004 where I'd run mkngff and viewed the images using previous build of ZarrReader - https://github.com/IDR/idr-metadata/issues/637#issuecomment-1708133932 On this plate, images rendered fine - looks good.

Then I tried viewing a different Plate where I'd run mkngff at the same time but not viewed the images (no memo file created). Plate P144, e.g. http://localhost:1040/webclient/?show=image-696604

Memo file created just now...

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "08-34-48.864_mkngff"
2023-09-13 09:35:24,553 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/..zattrs.bfmemo (219313 bytes)
2023-09-13 09:35:24,553 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1694597666657] time[57896] tag[loci.formats.Memoizer.setId]
2023-09-13 09:35:24,553 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/.zattrs Series: 0

But this image is black, even with the rendering settings adjusted to be bright:

Screenshot 2023-09-13 at 10 45 40

Also, the black images are rendering very slowly - ~25 seconds, compared to Plate P170 above where the images are rendering correctly at "normal" speeds (less than 0.5 secs).

will-moore commented 12 months ago

Discussed this morning - improved logging, comparison between memo file generation on import and after mkngff.

Update today's build of ZarrReader on idr-testing:omeoreadwrite...

https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/539/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Then tried viewing 3rd Plate from idr0025: http://localhost:1080/webclient/?show=image-3260835

3-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IContainer
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery(omero.api._IQueryTie@ab037ce6)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin(omero.api._IAdminTie@8040fd4d)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript(omero.api._IScriptTie@b8b56855)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig(omero.api._IConfigTie@a6cf999d)
2023-09-15 09:41:18,674 INFO  [                      omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig
2023-09-15 09:41:23,692 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:23,800 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/1
2023-09-15 09:41:23,995 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,106 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/2
2023-09-15 09:41:24,190 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,300 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/3
2023-09-15 09:41:24,517 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,743 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 5 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/1
2023-09-15 09:41:24,803 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,955 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 6 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/2
2023-09-15 09:41:25,212 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:25,368 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 7 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/3
2023-09-15 09:41:25,674 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:25,830 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 9 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/1
2023-09-15 09:41:25,899 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,006 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 10 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/2
2023-09-15 09:41:26,122 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,232 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 11 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/3
2023-09-15 09:41:26,544 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,794 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 13 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/1
2023-09-15 09:41:26,863 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,975 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 14 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/2
2023-09-15 09:41:27,038 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:27,113 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 15 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/3
2023-09-15 09:41:27,576 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,741 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 17 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/1
2023-09-15 09:41:27,814 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,925 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 18 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/2
2023-09-15 09:41:27,993 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:28,099 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 19 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/3
2023-09-15 09:41:28,508 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,811 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 21 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/1
2023-09-15 09:41:28,874 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,983 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 22 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/2
2023-09-15 09:41:29,119 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:29,229 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 23 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/3
2023-09-15 09:41:29,509 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,677 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 25 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/1
2023-09-15 09:41:29,753 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,868 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 26 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/2
2023-09-15 09:41:29,940 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:30,101 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 27 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/3
2023-09-15 09:41:30,410 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,572 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 29 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/1
2023-09-15 09:41:30,637 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,706 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 30 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/2
2023-09-15 09:41:30,764 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,971 INFO  [              loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 31 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e

...
23-09-15 09:50:33,759 INFO  [                      omero.cmd.SessionI] (l.Server-8) Unregistered servant:fa0664ae-6693-4e84-ad2a-c48335260413/472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer(omero.api._IContainerTie@be549585)
2023-09-15 09:50:33,759 INFO  [                      omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: 472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo9156189850653740893
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694771435991] time[104] tag[loci.formats.Memoizer.saveMemo]
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
2023-09-15

will-moore commented 12 months ago

First attempt to view the Image above gave this in webclient...

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1008, in getRenderingDefId
    return _M_omero.api.RenderingEngine._op_getRenderingDefId.invoke(self, ((), _ctx))
Ice.ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero

But on next attempt it worked and Image is NOT black!

Screenshot 2023-09-15 at 10 55 33

will-moore commented 12 months ago

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "15-26-46.818_mkngff"
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694516674666] time[244964] tag[loci.formats.Memoizer.setId]
2023-09-12 11:08:39,631 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-13 23:00:16,310 DEBUG [                   loci.formats.Memoizer] (.Server-13) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2018-01/24/14-20-58.331_mkngff/8c168760-7bc0-4692-90b1-c774711e7dd8.zarr/OME/.METADATA.ome.xml.bfmemo (666953 bytes)
--
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0

712022 ms is 12 minutes.

Previous plates on idr0125 have taken 95 minutes for Plate 1 and 4 minutes also on Plate 3 on idr-testing server.

will-moore commented 12 months ago

Comparing regular import (no chunks) and timing of memo file creation with after mkngff...

On idr0125-pilot (we already have NGFF data without chunks) - update ZarrReader as above...

cd /ngff/idr0025

for dir in *; do
  omero import --transfer=ln_s --depth=100 --name="${dir/.ome.zarr/}" --skip=all "$dir" --file "/tmp/$dir.log"  --errs "/tmp/$dir.err";
done

But err logs have:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-15 10:39:17,672 294        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-15 10:39:17,691 313        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 995067a5752310c81d89accc34337b4b1d0624df date: 20230915-0010
2023-09-15 10:39:17,753 375        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-15 10:39:17,754 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-15 10:39:17,755 377        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-15 10:39:17,758 380        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-15 10:39:18,155 777        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-09-15 10:40:44,670 87292      [      main] INFO      ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 78529ms. (86515ms total) [0 unknowns]
2023-09-15 10:40:44,722 87344      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-15 10:40:45,306 87928      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-15 10:40:45,960 88582      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-15 10:40:45,973 88595      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found

will-moore commented 12 months ago

Reverted ZarrReader jar to OMEZarrReader-0.3.1-jar-with-dependencies.jar on idr0125-pilot and re-ran import of idr0025 plates above...

e.g.

2023-09-15 11:36:42,662 1321330    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287669

==> Summary
2509 files uploaded, 1 fileset, 1 plate created, 384 images imported, 0 errors in 0:21:56.370

Blank images (no chunks) are viewable.

I see several "saved memo" events in the logs. Times are 1.5 - 3.5 seconds.

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "11-14-46.693/10x"
2023-09-15 11:36:08,898 DEBUG [                   loci.formats.Memoizer] (2-thread-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503832 bytes)
2023-09-15 11:36:08,899 DEBUG [                   loci.formats.Memoizer] (2-thread-1) start[1694777765366] time[3532] tag[loci.formats.Memoizer.setId]
2023-09-15 11:36:09,000 INFO  [    o.s.blitz.repo.ManagedImportRequestI] (2-thread-1) File format: Zarr
--
2023-09-15 12:02:14,851 DEBUG [                   loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,851 DEBUG [                   loci.formats.Memoizer] (l.Server-6) start[1694779333111] time[1740] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,852 INFO  [                ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 300
--
2023-09-15 12:02:14,899 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,899 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694779333111] time[1787] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,899 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 132

Rendering of image planes in webclient takes 200-300 ms.

Now update symlink as omero-server user... using https://github.com/IDR/idr-metadata/issues/647#issuecomment-1697382761

cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
rm -rf "10x images plate 1.ome.zarr"
ln -s /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr "10x images plate 1.ome.zarr"

Image can be viewed as normal - "Save to All" generates thumbs for the Plate.

Then removed the memo file...

rm -rf "/data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo"

Trying to view the Image in webclient:

  serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml

will-moore commented 12 months ago

EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908

Will try mkngff on idr0025/Plate 1 (with deleted memo file) from above...

(mkngff) bash-4.2$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287669 "/bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr" > "5287669.sql"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15 // 11-14-46.693 for fileset 5287669
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr -> /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr

$ psql -U omero -d idr -h $DBHOST -f 5287669.sql
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        5287719
(1 row)
COMMIT

Viewing image...

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
}

Blitz.log

2023-09-15 13:31:52,866 INFO  [         ome.security.basic.EventHandler] (l.Server-3)  Auth:    user=52,group=3,event=326848895(User),sess=6abc8eea-4910-4f1b-b57d-cf8b244189d8
2023-09-15 13:31:53,391 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1694784712636] time[755] tag[loci.formats.Memoizer.setId]
2023-09-15 13:31:53,391 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
2023-09-15 13:31:53,391 ERROR [                ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor340.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
        at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
        at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
        at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
        at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)

Checked that we only have a single unique original file path/name:

idr=> select id from OriginalFile where path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/' and name = 'METADATA.ome.xml';
    id    
----------
 52318508
(1 row)

will-moore commented 12 months ago

EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908

Try on Plate 2

omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287670 "/bia-integrator-data/S-BIAD846/52304cdf-4eba-4f0a-84b1-690e0d66add9/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr" > 5287670.sql

(took 8 minutes)...

$ psql -U omero -d idr -h $DBHOST -f 5287670.sql 
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        5287720
(1 row)
COMMIT

Not working...

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-09/15/11-36-48.798_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml
}

will-moore commented 12 months ago

EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908

mkngff hasn't been working today on NGFF Filesets. Going to try again on original .pattern file data...

On idr0125-pilot... still trying with OMEZarrReader-0.3.1-jar-with-dependencies.jar Try with some Plates from idr0011...

Try with first 4 Plates from first screen...

idr0011.csv

idr0011/Plate1-Blue-A.ome.zarr,S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f,21689
idr0011/Plate1-Blue-B.ome.zarr,S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1,21687
idr0011/Plate1-Green-A.ome.zarr,S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b,21688
idr0011/Plate1-Green-B.ome.zarr,S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b,21685

$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-56-03.328 for fileset 21689
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr -> /bia-integrator-data/S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-54-46.325 for fileset 21687
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr -> /bia-integrator-data/S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-55-31.533 for fileset 21688
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr -> /bia-integrator-data/S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-43-55.929 for fileset 21685
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr -> /bia-integrator-data/S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr

bash-4.2$ for r in $(cat $IDRID.csv); do
>   fsid=$(echo $r | cut -d',' -f3)
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
> done
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287721
(1 row)

COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287722
(1 row)

COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287723
(1 row)

COMMIT
UPDATE 46
BEGIN
 mkngff_fileset 
----------------
        5287724
(1 row)

COMMIT

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/OME/METADATA.ome.xml
}

Something wrong with omero-mkngff? All above today was testing with https://github.com/IDR/omero-mkngff/pull/9 (which shouldn't behave any differently). But lets revert to version I've been using mostly...

 pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'

Try with 2 more plates from idr0011...

idr0011/Plate1-Red-A.ome.zarr,S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d,21692
idr0011/Plate1-Red-B.ome.zarr,S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086,21686

$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3);   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"; done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 14-00-14.412 for fileset 21692
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr -> /bia-integrator-data/S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-48-45.381 for fileset 21686
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr -> /bia-integrator-data/S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr

$ for r in $(cat $IDRID.csv); do   fsid=$(echo $r | cut -d',' -f3);   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"; done
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287725
(1 row)
COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5287726
(1 row)
COMMIT

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr/OME/METADATA.ome.xml
}

will-moore commented 11 months ago

EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908

Looking at the Blitz log for the last exception above (viewing http://localhost:1080/webclient/?show=well-1227572 from idr0011/Plate1-Red-A.ome.zarr I see...

Caused by: java.lang.IllegalArgumentException: Invalid index: -1
    at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
    at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
    at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
    at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)
    at loci.formats.FormatReader.setId(FormatReader.java:1443)
    at loci.formats.ImageReader.setId(ImageReader.java:849)
2023-09-18 07:58:56,503 INFO  [                 org.perf4j.TimingLogger] (l.Ser

Looking at the code... https://github.com/ome/bioformats/blob/bc1fcddd5329d900853d55c6f8378e9235a26a53/components/formats-api/src/loci/formats/FormatReader.java#L960

Here, core.size() is returning 0 so that we get coreIndexToSeries(-1) being called, which throws exception above.

will-moore commented 11 months ago

Discussed Friday's testing at IDR meeting today - I realise that the released OMEZarrReader-0.3.1-jar-with-dependencies.jar is likely failing for mkngff workflow because it is missing one of the currently open PRs: This one (#64), #58, #53 or #20.

So I will return to testing mkngff workflow with the merge-ci build...

will-moore commented 11 months ago

Using todays merge-ci build of ZarrReader: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/617/console

2023-09-18 00:06:16,745 [   scc.merge] INFO  Repository: ome/ZarrReader
2023-09-18 00:06:16,745 [   scc.merge] INFO  Excluded PRs:
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 58 dgault 'Add new option to list pixels in getUsedFiles' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO    - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-09-18 00:06:16,745 [   scc.merge] INFO  Already up-to-date.
2023-09-18 00:06:16,746 [   scc.merge] INFO  
2023-09-18 00:06:16,746 [   scc.merge] INFO  Merged PRs:
2023-09-18 00:06:16,746 [   scc.merge] INFO    - PR 62 joshmoore 'Update README.md'
2023-09-18 00:06:16,746 [   scc.merge] INFO    - PR 64 dgault 'ZarrReader performance improvements'

From https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/default/lastSuccessfulBuild/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Testing memo file generation by viewing previously unviewed mkngff plate from idr0033:

will-moore commented 11 months ago

Viewing image from 2nd plate: http://localhost:1080/webclient/?show=image-3230447 having checked the logs for memo file generation for that fileset to confirm that we haven't yet generated memo file..

grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.2 | grep -A 2 "2017-03/07/16-50-40.721_mkngff"

will-moore commented 11 months ago

Checking on progress of memo file generation (started 3 hours ago)...

[wmoore@test120-omeroreadwrite ~]$ grep "Opening zarr for series" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2023-09-18 12:35:03,019 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,218 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/1
2023-09-18 12:35:03,289 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,392 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/2
2023-09-18 12:35:03,460 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,573 INFO  [              loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/3
2023-09-18 12:44:28,495 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,572 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,691 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:44:28,748 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,816 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:44:28,818 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,860 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:44:28,921 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:44:28,922 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,924 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:29,026 INFO  [              loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:44:29,027 INFO  [              loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:52:38,383 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,550 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:52:38,615 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,733 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:52:38,781 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,896 INFO  [              loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:56:49,997 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,012 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,154 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:56:50,154 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:56:50,202 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,202 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,278 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:56:50,303 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:56:50,325 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,326 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,400 INFO  [              loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:56:50,440 INFO  [              loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:40:54,300 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,415 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:40:54,463 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,532 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:40:54,579 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,683 INFO  [              loci.formats.FormatHandler] (.Server-11) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:56:45,188 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,188 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,629 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:56:45,638 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:56:45,709 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,716 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,849 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:56:45,851 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:56:45,914 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,914 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,979 INFO  [              loci.formats.FormatHandler] (.Server-10) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:56:46,042 INFO  [              loci.formats.FormatHandler] (.Server-13) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 14:43:00,991 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,140 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 14:43:01,203 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,308 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 14:43:01,436 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,551 INFO  [              loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3

All these logs are from the first field of A/1 at A/1/0.

Attempting to view a different Well or Field from that Plate doesn't show any more similar logs. Presumably some flag indicates that memo file generation is in progress already...?

will-moore commented 11 months ago

Images from Plate 2 are finally visible today. Strange in logs below that we have logs from several "saved memo" events overnight: NB: add 1 hour for BST: 23:04:42, 01:52:22, 02:49:16, 04:15:45, 04:15:46, 38519781 ms is 10.6 hours. 47240506 ms is 13 hours

grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "16-50-40.721_mkngff/e45c"

2023-09-18 23:04:42,825 DEBUG [                   loci.formats.Memoizer] (l.Server-0) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,825 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1695039763044] time[38519781] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,826 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-18 23:04:42,841 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,841 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1695040253748] time[38029092] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,841 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-18 23:04:42,973 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,973 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1695040759427] time[37523545] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,973 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,626 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,626 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1695041102119] time[47240506] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,626 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,822 DEBUG [                   loci.formats.Memoizer] (.Server-11) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,822 DEBUG [                   loci.formats.Memoizer] (.Server-11) start[1695043675269] time[44667552] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,822 INFO  [                ome.io.nio.PixelsService] (.Server-11) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,856 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,856 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1695041096479] time[47246376] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,856 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 02:49:16,679 DEBUG [                   loci.formats.Memoizer] (.Server-10) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,679 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1695044671046] time[47085633] tag[loci.formats.Memoizer.setId]
2023-09-19 02:49:16,680 INFO  [                ome.io.nio.PixelsService] (.Server-10) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0

--
2023-09-19 02:49:16,950 DEBUG [                   loci.formats.Memoizer] (.Server-13) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,950 DEBUG [                   loci.formats.Memoizer] (.Server-13) start[1695044686175] time[47070775] tag[loci.formats.Memoizer.setId]
2023-09-19 02:49:16,950 INFO  [                ome.io.nio.PixelsService] (.Server-13) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 02:49:16,958 DEBUG [                   loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,958 INFO  [                 org.perf4j.TimingLogger] (2-thread-2) start[1695091756958] time[0] tag[omero.call.success.ome.services.RenderingBean.close]
2023-09-19 02:49:16,959 INFO  [        ome.services.util.ServiceHandler] (2-thread-2)  Rslt:    null
--
2023-09-19 04:15:45,698 DEBUG [                   loci.formats.Memoizer] (.Server-14) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,699 DEBUG [                   loci.formats.Memoizer] (.Server-14) start[1695066384572] time[30561126] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,699 INFO  [                ome.io.nio.PixelsService] (.Server-14) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,837 DEBUG [                   loci.formats.Memoizer] (.Server-20) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,837 DEBUG [                   loci.formats.Memoizer] (.Server-20) start[1695058260700] time[38685136] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,837 INFO  [                ome.io.nio.PixelsService] (.Server-20) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,935 DEBUG [                   loci.formats.Memoizer] (l.Server-8) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,935 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1695051155880] time[45790055] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,936 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,949 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,949 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1695050732463] time[46213485] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,949 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,090 DEBUG [                   loci.formats.Memoizer] (.Server-12) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,090 DEBUG [                   loci.formats.Memoizer] (.Server-12) start[1695051103810] time[45842280] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,090 INFO  [                ome.io.nio.PixelsService] (.Server-12) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,210 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,210 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1695051108480] time[45837730] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,210 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,253 DEBUG [                   loci.formats.Memoizer] (.Server-16) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,253 DEBUG [                   loci.formats.Memoizer] (.Server-16) start[1695050749658] time[46196595] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,253 INFO  [                ome.io.nio.PixelsService] (.Server-16) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,377 DEBUG [                   loci.formats.Memoizer] (.Server-15) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,377 DEBUG [                   loci.formats.Memoizer] (.Server-15) start[1695051598118] time[45348259] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,377 INFO  [                ome.io.nio.PixelsService] (.Server-15) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 9

will-moore commented 11 months ago

As discussed after standup this morning:

I will look at cli or python scripting options for triggering memo file generation instead of repeated refreshing of webclient.
David will temp hard-code the logging of import scanning at ERROR so that we can see the output from Bio-Formats to debug the "No imports found"

2023-09-15 10:39:17,758 380        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO

will-moore commented 11 months ago

With OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar from David with ERROR logging, updated onto idr-testing, restart etc.

On idr-testing, as omero-server user, made a local copy of the first Plate from idr0011 for testing...

$ pwd
/opt/omero/server/ngff/idr0011
$ cp -R /bia-integrator-data/S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr .

Did regular import of Plates with chunks:omero import --depth=10 f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr

Log file lists every image etc for each chunk. E.g. .zarr/A/1/0/0/0/1/7/0/0:

...
2023-09-19 10:33:16,010 309337     [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/7/0/2
2023-09-19 10:33:16,010 309337     [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/0
2023-09-19 10:33:16,010 309337     [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 10:33:16,010 309337     [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 10:33:16,010 309337     [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 10:33:16,011 309338     [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 10:33:16,013 309340     [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 10:33:16,412 309739     [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/A/1/0/0/0/1/7/0/0
2023-09-19 10:33:16,412 309739     [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 10:33:16,412 309739     [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML

...
2023-09-19 11:50:42,622 4955949    [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 11:50:42,622 4955949    [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 11:50:42,622 4955949    [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 11:50:42,622 4955949    [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 11:50:42,623 4955950    [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 11:50:42,987 4956314    [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/A/4/0/1/0/1/13/0/0
2023-09-19 11:50:42,987 4956314    [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 11:50:42,987 4956314    [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML

...
2023-09-19 20:11:58,424 35031751   [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/0
2023-09-19 20:11:58,424 35031751   [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 20:11:58,424 35031751   [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 20:11:58,424 35031751   [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 20:11:58,424 35031751   [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 20:11:58,425 35031752   [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 20:11:58,790 35032117   [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/C/8/0/2/0/2/3/0/0
2023-09-19 20:11:58,790 35032117   [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 20:11:58,790 35032117   [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML

Cancelled after 20 hours, processing 24th image out of 48.

will-moore commented 11 months ago

Try to retest the https://github.com/ome/ZarrReader/pull/64#issuecomment-1721072253 "No imports found" above.

Updated today's OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar (with ERROR logging) onto idr0125-pilot...

Then

$ cd /ngff/idr0025
$ omero import --transfer=ln_s --skip=all --depth=10 "10x images plate 1.ome.zarr" --file /tmp/idr0025_20230919.log --err /tmp/idr0025_20230919.err

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-19 12:52:15,840 306        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-19 12:52:15,860 326        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 6371c790a70306bbfcf1365d64bad04c19fc538d date: 2023-09-19T09:40:05Z
2023-09-19 12:52:15,928 394        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-19 12:52:15,929 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-19 12:52:15,929 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-19 12:52:15,930 396        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-19 12:52:15,930 396        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-19 12:52:15,933 399        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-19 12:52:16,337 803        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-09-19 12:52:16,453 919        [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-19 12:52:16,460 926        [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-19 12:52:16,468 934        [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-09-19 12:52:17,336 1802       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-19 12:52:17,353 1819       [      main] ERROR                loci.formats.FormatHandler - ZarrReader generating group keys
2023-09-19 12:52:17,353 1819       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-09-19 12:52:46,916 31382      [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-09-19 12:52:46,920 31386      [      main] ERROR                loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-09-19 12:52:46,960 31426      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-19 12:52:46,960 31426      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:46,981 31447      [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:46,982 31448      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-19 12:52:46,982 31448      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1
2023-09-19 12:52:46,990 31456      [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:46,991 31457      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-19 12:52:46,991 31457      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:46,991 31457      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2
2023-09-19 12:52:47,006 31472      [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:47,007 31473      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3
2023-09-19 12:52:47,007 31473      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:47,008 31474      [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/3
2023-09-19 12:52:47,022 31488      [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:47,022 31488      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-19 12:52:47,022 31488      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-19 12:52:47,022 31488      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-19 12:52:47,022 31488      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3

...
2023-09-19 12:52:47,093 31559      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-19 12:52:47,093 31559      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-19 12:52:47,093 31559      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-19 12:52:47,093 31559      [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 12:52:47,139 31605      [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 12:52:47,170 31636      [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 12:52:47,320 31786      [      main] INFO      ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 30872ms. (30983ms total) [0 unknowns]
2023-09-19 12:52:47,367 31833      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-19 12:52:47,872 32338      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-19 12:52:48,825 33291      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-19 12:52:48,844 33310      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-19 12:52:48,845 33311      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-19 12:52:48,845 33311      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-19 12:52:48,845 33311      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-19 12:52:48,845 33311      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-19 12:52:48,845 33311      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found

will-moore commented 11 months ago

Testing performance of memo file generation on Plates from idr0011 on idr-testing...

http://localhost:1080/webclient/?show=well-1227779

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "888_mkngff/5fbc36b"
2023-09-20 09:23:46,281 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo (6990617 bytes)
2023-09-20 09:23:46,281 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1695201778449] time[47832] tag[loci.formats.Memoizer.setId]
2023-09-20 09:23:46,282 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/METADATA.ome.xml Series: 1

47832 is 48 seconds

Deleted memo file, viewed image again and it got recreated

[wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo

logs

2023-09-20 09:31:01,247 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo (6990617 bytes)
2023-09-20 09:31:01,247 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1695202188874] time[72373] tag[loci.formats.Memoizer.setId]
2023-09-20 09:31:01,247 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/METADATA.ome.xml Series: 1

72373 is 72 seconds (bit slower)?

Trying with next plate http://localhost:1080/webclient/?show=image-2850468 - creating memo, deleting then creating again gives similar times for both: 49 secs and 52 secs:

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "539_mkngff/196473f"
2023-09-20 09:33:49,246 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/.METADATA.ome.xml.bfmemo (6987787 bytes)
2023-09-20 09:33:49,246 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1695202379756] time[49490] tag[loci.formats.Memoizer.setId]
2023-09-20 09:33:49,246 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-20 09:35:33,753 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/.METADATA.ome.xml.bfmemo (6987787 bytes)
2023-09-20 09:35:33,753 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1695202481531] time[52222] tag[loci.formats.Memoizer.setId]
2023-09-20 09:35:33,753 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/METADATA.ome.xml Series: 0

will-moore commented 11 months ago

Viewing plate 1 from idr0025 - triggered a memo file generation (without deleting previous one)...

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 ".590_mkngff/52304cd"
2023-09-20 09:51:18,564 DEBUG [                   loci.formats.Memoizer] (l.Server-0) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/.METADATA.ome.xml.bfmemo (876328 bytes)
2023-09-20 09:51:18,564 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1695203292938] time[185625] tag[loci.formats.Memoizer.setId]
2023-09-20 09:51:18,564 INFO  [                ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml Series: 0

185625 ms is 3 minutes

Plate 2 memo file already exists -rw-rw-r--. 1 omero-server omero-server 551K Sep 15 10:13 .METADATA.ome.xml.bfmemo Previous time took 11 mins to generate:

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.1 | grep -A 2 "4.747_mkngff/72cc29"
2023-09-15 10:13:12,445 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 10:13:12,445 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1694772108579] time[683865] tag[loci.formats.Memoizer.setId]
2023-09-15 10:13:12,445 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 76

Deleted...

[wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "4.747_mkngff/72cc29"
2023-09-20 10:02:39,729 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (876328 bytes)
2023-09-20 10:02:39,729 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1695203993000] time[166728] tag[loci.formats.Memoizer.setId]
2023-09-20 10:02:39,729 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 0

166728 is 2.7 minutes 👍

will-moore commented 11 months ago

Looks like ZarrReader is assuming every Well is filled - getting exceptions from Plates with missing Wells: https://github.com/IDR/idr-metadata/issues/644#issuecomment-1727543657

will-moore commented 11 months ago

Updated idr0125-pilot... (don't want to restart idr-testing server during long-running omero mkngff sql generation)...

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/545/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
cp OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
...
sudo service omero-server restart

Test viewing of sparse Plate in idr0004 (previously generated):

Screenshot 2023-09-21 at 11 49 16

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "185_mkngff/103d9428" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 10:34:53,599 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/..zattrs.bfmemo (171808 bytes)
2023-09-21 10:34:53,599 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1695292350587] time[143012] tag[loci.formats.Memoizer.setId]
2023-09-21 10:34:53,599 INFO  [                ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/.zattrs Series: 0

143012 ms is 2.4 minutes

will-moore commented 11 months ago

Tried import again on idr0125-pilot...

cd /ngff/idr0025
omero import --transfer=ln_s --skip=all --depth=10 "10x images plate 1.ome.zarr" --file /tmp/idr0025_20230921.log --err /tmp/idr0025_20230921.err

err

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-21 12:14:58,287 286        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-21 12:14:58,308 307        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 237c1009f320c633bbdec5b0937647a320c5c250 date: 20230921-0010
2023-09-21 12:14:58,384 383        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-21 12:14:58,386 385        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-21 12:14:58,386 385        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-21 12:14:58,386 385        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-21 12:14:58,387 386        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-21 12:14:58,392 391        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-21 12:14:58,772 771        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-09-21 12:14:58,849 848        [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-21 12:14:58,855 854        [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:58,862 861        [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-09-21 12:14:59,533 1532       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:59,549 1548       [      main] ERROR                loci.formats.FormatHandler - ZarrReader generating group keys
2023-09-21 12:14:59,556 1555       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-09-21 12:14:59,728 1727       [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-09-21 12:14:59,736 1735       [      main] ERROR                loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-09-21 12:14:59,761 1760       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-21 12:14:59,761 1760       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,775 1774       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,775 1774       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-21 12:14:59,775 1774       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1
2023-09-21 12:14:59,776 1775       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,776 1775       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-21 12:14:59,776 1775       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,776 1775       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/3
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,777 1776       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-21 12:14:59,778 1777       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-21 12:14:59,778 1777       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
...
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/1
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/2
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/3
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/0
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/1
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/2
2023-09-21 12:14:59,817 1816       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/3
...
2023-09-21 12:14:59,839 1838       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/0
2023-09-21 12:14:59,839 1838       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-21 12:14:59,839 1838       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-21 12:14:59,840 1839       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-21 12:14:59,840 1839       [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-21 12:14:59,888 1887       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-21 12:14:59,917 1916       [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-21 12:14:59,938 1937       [      main] ERROR                loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-09-21 12:14:59,939 1938       [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:59,944 1943       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0/.zarray
2023-09-21 12:14:59,944 1943       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1/.zarray
2023-09-21 12:14:59,944 1943       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2/.zarray
...
2023-09-21 12:15:00,031 2030       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/H/.zgroup
2023-09-21 12:15:00,031 2030       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/.zattrs
2023-09-21 12:15:00,031 2030       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/.zgroup
2023-09-21 12:15:00,031 2030       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-21 12:15:00,031 2030       [      main] ERROR                loci.formats.FormatHandler - ZarrReader returning list of used files of size: 972
2023-09-21 12:15:00,104 2103       [      main] INFO      ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 1213ms. (1333ms total) [0 unknowns]
2023-09-21 12:15:00,149 2148       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-21 12:15:00,616 2615       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-21 12:15:01,070 3069       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-21 12:15:01,084 3083       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found

will-moore commented 11 months ago

Repeating import with today's merge build of ZarrReader on idr0125-pilot...

repeating https://github.com/ome/ZarrReader/pull/64#issuecomment-1721072253

Import of idr0025 Plate 1 (without chunks): looking good...

2023-09-25 14:57:57,059 40298      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/0
2023-09-25 14:57:57,059 40298      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-25 14:57:57,059 40298      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-25 14:57:57,059 40298      [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-25 14:57:57,059 40298      [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-25 14:57:57,109 40348      [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-25 14:57:57,138 40377      [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-25 14:57:57,156 40395      [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-25 14:57:57,215 40454      [      main] ERROR                loci.formats.FormatHandler - ZarrReader returning list of used files of size: 2509
2023-09-25 14:58:00,001 43240      [      main] INFO      ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 1 group(s) with 1 call(s) to setId in 27753ms. (41061ms total) [0 unknowns]
2023-09-25 14:58:01,603 44842      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-25 14:58:05,203 48442      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-25 14:58:06,700 49939      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-25 14:58:06,721 49960      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-25 14:58:06,721 49960      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-25 14:58:06,722 49961      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-25 14:58:06,722 49961      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-25 14:58:06,722 49961      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-25 14:58:06,722 49961      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-09-25 14:58:06,892 50131      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
...

Image:14993619
Image:14993620
Image:14993621
Image:14993622

==> Summary
2509 files uploaded, 1 fileset, 1 plate created, 672 images imported, 0 errors in 0:24:21.111

imagesc-bot commented 11 months ago

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/memory-error-on-zarr-import-on-omero/86684/2

snoopycrimecop commented 11 months ago

Conflicting PR. Removed from build BIOFORMATS-push#635. See the console output for more details. Possible conflicts:

PR #58 dgault 'Add new option to list pixels in getUsedFiles'
- src/loci/formats/in/ZarrReader.java

--conflicts

snoopycrimecop commented 11 months ago

Conflicting PR. Removed from build BIOFORMATS-push#636. See the console output for more details. Possible conflicts:

PR #58 dgault 'Add new option to list pixels in getUsedFiles'
- src/loci/formats/in/ZarrReader.java

~~--conflicts~~ Conflict resolved in build BIOFORMATS-push#637. See the console output for more details.

will-moore commented 11 months ago

As discussed this morning, those getUsedFiles changes fix the errors reported at https://github.com/IDR/idr-utils/pull/56#issuecomment-1746821150. However, it is also expected that they will break the regular import of NGFF data (ignoring chunks).

Want to test whether the released ZarrReader (without this PR or getUsedFiles changes) works when we import an image without chunks.

Updated idr-testing:omeroreadwrite server with released jar... and restarted

bash-4.2$ cp OMEZarrReader-0.3.1-jar-with-dependencies.jar /opt/omero/server/OMERO.server/lib/server/
bash-4.2$ cp OMEZarrReader-0.3.1-jar-with-dependencies.jar /opt/omero/server/OMERO.server/lib/client/

Download metadata only image from https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120.html

aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr .
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/.zattrs to ./.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/.zgroup to 0/.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/.zgroup to ./.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/.zattrs to 0/.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/0/.zarray to 0/0/.zarray
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/1/.zarray to 0/1/.zarray
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/.zgroup to OME/.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/.zattrs to OME/.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/METADATA.ome.xml to OME/METADATA.ome.xml

Import into idr-testing...

$ (venv3) [wmoore@test120-omeroreadwrite idr0051]$ omero import --depth=10 -d 3357 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip
Using OMERO.java-5.6.9-ice36
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Oct 05, 2023 11:21:28 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
2023-10-05 11:21:28,841 374        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2023-10-05 11:21:28,859 392        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2023-10-05 11:21:28,914 447        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-05 11:21:29,257 790        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-10-05 11:21:29,716 1249       [      main] INFO      ome.formats.importer.ImportCandidates - 9 file(s) parsed into 1 group(s) with 1 call(s) to setId in 455ms. (459ms total) [0 unknowns]
2023-10-05 11:21:29,758 1291       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-05 11:21:30,259 1792       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-05 11:21:30,956 2489       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-05 11:21:30,978 2511       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.95.1.el7.x86_64
2023-10-05 11:21:31,604 3137       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:3357
2023-10-05 11:21:31,647 3180       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:3357
2023-10-05 11:21:31,651 3184       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-10-05 11:21:51,999 23532      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-10-05 11:21:52,021 23554      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml...
2023-10-05 11:21:52,120 23653      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
2023-10-05 11:21:52,225 23758      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
2023-10-05 11:21:52,270 23803      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs...
2023-10-05 11:21:52,335 23868      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs
2023-10-05 11:21:52,486 24019      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs
2023-10-05 11:21:52,511 24044      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup...
2023-10-05 11:21:52,562 24095      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup
2023-10-05 11:21:52,696 24229      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup
2023-10-05 11:21:52,727 24260      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs...
2023-10-05 11:21:52,782 24315      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs
2023-10-05 11:21:52,910 24443      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs
2023-10-05 11:21:52,937 24470      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup...
2023-10-05 11:21:52,995 24528      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup
2023-10-05 11:21:53,053 24586      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup
2023-10-05 11:21:53,139 24672      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray...
2023-10-05 11:21:53,189 24722      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray
2023-10-05 11:21:53,262 24795      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray
2023-10-05 11:21:53,291 24824      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray...
2023-10-05 11:21:53,348 24881      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray
2023-10-05 11:21:53,519 25052      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray
2023-10-05 11:21:53,548 25081      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs...
2023-10-05 11:21:53,599 25132      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs
2023-10-05 11:21:53,861 25394      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs
2023-10-05 11:21:53,889 25422      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup...
2023-10-05 11:21:53,937 25470      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup
2023-10-05 11:21:54,002 25535      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup
2023-10-05 11:21:54,810 26343      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-05 11:21:55,105 26638      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 51578493
2023-10-05 11:22:02,441 33974      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 51578493
2023-10-05 11:22:49,139 80672      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 51578493
2023-10-05 11:22:50,921 82454      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 51578493
2023-10-05 11:22:51,101 82634      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 51578493
2023-10-05 11:22:51,149 82682      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 51578493
2023-10-05 11:22:51,312 82845      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
Image:14874782
Other imported objects:
Fileset:6312209

==> Summary
9 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:01:20.030

This image shows up as black as expected (no chunks) but is viewable. But also, the mkngff version of the same image (https://github.com/IDR/idr-utils/pull/56#issuecomment-1746831543) is also still visible using OMEZarrReader-0.3.1-jar-with-dependencies.jar, without getUsedFile changes.

will-moore commented 11 months ago

Strangely, the mkngff image from idr0051 above that is viewable with OMEZarrReader-0.3.1-jar-with-dependencies.jar appears to be the exception. Other mkngff Images and Plates are not working with that jar.

2023-10-05 11:48:05,961 INFO  [                loci.formats.ImageReader] (l.Server-0) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zar
r/OME/METADATA.ome.xml
2023-10-05 11:48:07,028 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1696506484058] time[2970] tag[loci.formats.Memoizer.setId]
2023-10-05 11:48:07,029 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b1
14-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-05 11:48:07,030 ERROR [                ome.io.nio.PixelsService] (l.Server-0) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697
900e39.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)

Try to import the original copy of the idr0051 image from s3 with chunks...

$ omero import --depth=10 -d 3357 /bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr
...

...taking a long time to walk all chunks on s3....

will-moore commented 11 months ago

Still with released ZarrReader 3.1, lets try mkngff without Directories or chunks on a single Plate 1-23 from idr0010 (still unchanged on idr-testing).

$ psql -U omero -d idr -h $DBHOST -f 20751.sql
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6312259
(1 row)
COMMIT

(venv3) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 20751 "/bia-integrator-data/S-BIAD885/0046b0d0-f20b-4482-84b1-4b2b154865fd/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr -> /bia-integrator-data/S-BIAD885/0046b0d0-f20b-4482-84b1-4b2b154865fd/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr

http://localhost:1080/webclient/?show=well-1033384 ...

This gives a ResourceError.

will-moore commented 11 months ago

Updating to OMEZarrReader_b558_p635.jar from https://github.com/IDR/idr-utils/pull/56#issuecomment-1746821150 fixes viewing of idr0010 plate:

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "102_mkngff/e7bdccbf" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-10-05 15:01:49,707 DEBUG [                   loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (2316371 bytes)
2023-10-05 15:01:49,707 DEBUG [                   loci.formats.Memoizer] (l.Server-6) start[1696517474813] time[634894] tag[loci.formats.Memoizer.setId]
2023-10-05 15:01:49,707 ERROR [              loci.formats.FormatHandler] (l.Server-1) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/A/9/0/2/0/1/0/0/0
--
2023-10-05 15:01:52,200 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (2316371 bytes)
2023-10-05 15:01:52,200 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1696517913517] time[198683] tag[loci.formats.Memoizer.setId]
2023-10-05 15:01:52,200 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 0

634894 ms is 10 minutes 198683 ms is 3 minutes

Screenshot 2023-10-05 at 16 22 10

will-moore commented 10 months ago

Testing different builds from jars build by @dgault... Copying jars and renaming...

rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/main-canonicalPR-getUsedFilesAllFiles/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_canonical_allfiles.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/main-canonicalPR-getUsedFilesNoPixels/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_canonical_nopixels.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/PerformancePR-getUsedFilesNoPixels/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_performance_nopixels.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/PerformancePR-getUsedFilesAllFiles/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_performance_allfiles.jar

Upload each to idr-testing:omeroreadwrite server. Copy them (as omero-server) to /opt/omero/server/OMERO.server/lib/server/ (and lib/server) and restart OMERO.

Test viewing of mkngff filesets - if previously viewed, this was using the ZarrReader version described at https://github.com/IDR/idr-metadata/issues/675#issuecomment-1755198551

With OMEZarrReader_canonical_nopixels.jar viewing previously viewed idr0054 images works. Viewing previously viewed idr0004 mkngff plate - I see lots of these logs in Blitz logs... This will take a long time to complete the plate...

i._IConfigTie@64645b60)
2023-10-18 11:48:25,453 INFO  [                      omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IConfig
2023-10-18 11:48:25,453 INFO  [                      omero.cmd.SessionI] (l.Server-8) Unregistered servant:8d74e6f3-5b9d-41ee-ac72-eb4839cc6b5e/e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IScript(omero.api._IScriptTie@7a387856)
2023-10-18 11:48:25,453 INFO  [                      omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IScript
2023-10-18 11:48:25,499 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/4/0/0
2023-10-18 11:48:25,748 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/5/0/0
2023-10-18 11:48:25,962 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/6/0/0
2023-10-18 11:48:26,222 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/7/0/0
2023-10-18 11:48:26,443 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/8/0/0
2023-10-18 11:48:26,663 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/9/0/0
2023-10-18 11:48:26,991 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/0/0/0
2023-10-18 11:48:27,239 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/1/0/0
2023-10-18 11:48:27,587 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/2/0/0
2023-10-18 11:48:27,999 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/3/0/0
2023-10-18 11:48:28,370 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/4/0/0
2023-10-18 11:48:28,599 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/5/0/0
2023-10-18 11:48:28,836 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/6/0/0
2023-10-18 11:48:29,052 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/7/0/0
2023-10-18 11:48:29,311 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/8/0/0
2023-10-18 11:48:29,636 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/9/0/0
2023-10-18 11:48:29,745 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/.zarray
2023-10-18 11:48:30,158 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/0/0/0/0
2023-10-18 11:48:30,501 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/0/1/0/0
2023-10-18 11:48:30,851 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07

Takes a long time, but eventually this plate is viewable.

Import as wmoore user (not inplace) from idr0054 on s3...

$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_nopixels.log --errs /tmp/idr0054_nopixels.err

No import found

Made metadata-only copy of image from idr0054 by copying image above, then deleting the chunks dir from each resolution. Also gave No import found.

Testing next with OMEZarrReader_canonical_allfiles.jar...

$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_allpixels.log --errs /tmp/idr0054_allpixels.err

13:26:22,957 488833     [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup...
2023-10-18 13:26:23,003 488879     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:26:23,176 489052     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:26:23,205 489081     [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml...
2023-10-18 13:26:23,258 489134     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:26:23,347 489223     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:26:30,470 496346     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 13:26:30,830 496706     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61515641
2023-10-18 13:26:36,885 502761     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61515641
2023-10-18 13:26:43,473 509349     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61515641
2023-10-18 13:26:47,910 513786     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61515641
2023-10-18 13:26:47,958 513834     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61515641
2023-10-18 13:26:47,998 513874     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61515641
2023-10-18 13:26:48,201 514077     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314491

==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:08:20.580

Import of metadata-only image from idr0054.

$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_allfiles_mo.log --errs /tmp/idr0054_allfiles_mo.err
...
2023-10-18 13:29:15,112 7278       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:29:15,236 7402       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:29:15,541 7707       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 13:29:15,637 7803       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61516095
2023-10-18 13:29:16,216 8382       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61516095
2023-10-18 13:29:21,080 13246      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61516095
2023-10-18 13:29:21,636 13802      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61516095
2023-10-18 13:29:21,691 13857      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61516095
2023-10-18 13:29:21,729 13895      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61516095
2023-10-18 13:29:21,926 14092      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314492

==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:11.501

Black image but viewing works without errors.

Testing OMEZarrReader_performance_nopixels.jar...

(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_allfiles.log --errs /tmp/idr0054_perf_allfiles.err

Import log repeats this a couple of times per second with all files being listed... Feels like infinite loop!

...
2023-10-18 13:50:33,051 119282     [      main] ERROR                loci.formats.FormatHandler - ZarrReader including performance PR and getUsedFiles changes (skips pixels)
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/.zattrs
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/.zgroup
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/.zattrs
2023-10-18 13:50:33,297 119528     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/.zgroup
2023-10-18 13:50:33,298 119529     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/.zarray
2023-10-18 13:50:33,298 119529     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
2023-10-18 13:50:33,298 119529     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/1
2023-10-18 13:50:33,298 119529     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/2
2023-10-18 13:50:33,299 119530     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/0
2023-10-18 13:50:33,299 119530     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/1
2023-10-18 13:50:33,299 119530     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/2
2023-10-18 13:50:33,299 119530     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/2/0
...
...finally...
2023-10-18 13:53:45,102 311333     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:53:45,102 311333     [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:53:45,104 311335     [      main] INFO      ome.formats.importer.ImportCandidates - 444 file(s) parsed into 0 group(s) with 433 call(s) to setId in 306519ms. (310440ms total) [0 unknowns]
2023-10-18 13:53:45,153 311384     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-18 13:53:45,692 311923     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-18 13:53:46,250 312481     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-18 13:53:46,269 312500     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.99.1.el7.x86_64
No imports found

Import metadata-only NGFF without chunks...

(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels_mo.log --errs /tmp/idr0054_perf_nopixels_mo.err
...
No import found

Finally, testing OMEZarrReader_performance_allfiles.jar...

(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels.log --errs /tmp/idr0054_perf_nopixels.err

...
2023-10-18 14:28:25,402 325332     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 14:28:25,659 325589     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 14:28:32,274 332204     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 14:28:32,464 332394     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61516644
2023-10-18 14:28:34,758 334688     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61516644
2023-10-18 14:28:41,556 341486     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61516644
2023-10-18 14:28:42,425 342355     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61516644
2023-10-18 14:28:42,481 342411     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61516644
2023-10-18 14:28:42,527 342457     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61516644
2023-10-18 14:28:42,722 342652     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314541

==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:05:35.200

metadata-only...

(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels_mo.log --errs /tmp/idr0054_perf_nopixels_mo.err

...
2023-10-18 14:32:54,532 9218       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 14:32:54,810 9496       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 14:32:54,915 9601       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61517100
2023-10-18 14:32:55,420 10106      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61517100
2023-10-18 14:33:00,874 15560      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61517100
2023-10-18 14:33:01,498 16184      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61517100
2023-10-18 14:33:01,544 16230      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61517100
2023-10-18 14:33:01,585 16271      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61517100
2023-10-18 14:33:01,785 16471      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314542

==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:14.078

Black image but import OK.

will-moore commented 10 months ago

As discussed with @dgault... The allfiles ZarrReaders above fail to create memo file for mkngff Filesets, although they can import NGFF OK. Question to test: can these allfiles ZarrReaders create a memo file for NGFF filesets created by regular import (if the memo file is deleted)?

We still are using OMEZarrReader_performance_allfiles.jar from yesterday...

Using the 4 images successfully imported yesterday with allfiles ZarrReaders (with/without performace PR, with/without chunks)

check server logs to get location of memo file... and delete...


(base) [wmoore@test120-omeroreadwrite ~]$ grep -A 2 "13-18-30.948/a0e1b9c0-5c07-" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.1 | grep -A 2 "saved memo"
2023-10-18 13:26:31,122 DEBUG [                   loci.formats.Memoizer] (2-thread-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/.0.bfmemo (46364 bytes)
2023-10-18 13:26:31,122 DEBUG [                   loci.formats.Memoizer] (2-thread-3) start[1697635590735] time[387] tag[loci.formats.Memoizer.setId]

(base) [wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/.0.bfmemo

The image was still viewable in the webclient and a new `.0.bfmemo` file has been created, although there's nothing new in the Blitz.log when searching with `13-18-30.948/a0e1b9c0-5c07-` as above.

(base) [wmoore@test120-omeroreadwrite ~]$ sudo ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/ total 40K drwxrwxr-x. 2 omero-server omero-server 31 Oct 19 09:05 . drwxrwxr-x. 3 omero-server omero-server 15 Oct 18 13:26 .. -rw-rw-r--. 1 omero-server omero-server 37K Oct 19 09:05 .0.bfmemo


Repeated with exactly the same behaviour for the other 3 images.

will-moore commented 10 months ago

I noticed a change in behaviour with this PR...

In the testing on idr-testing of various builds above https://github.com/ome/ZarrReader/pull/64#issuecomment-1768612367 I have imported from s3 bucket like this for a couple of tests...

omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr

With OMEZarrReader_canonical_allfiles.jar I got Fileset: 6314491. This has ManagedRepo paths like:

demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/2/0/21/0/0/0
demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4/0/10/0/0/0

With the OMEZarrReader_performance_allfiles.jar I got Fileset: 6314541. The clientpath is the same as above, but the ManagedRepo paths are much longer, since they contain the full path: (extra bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39 in each path):

demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/18/14-23-08.177/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1/0/8/0/1/0
demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/18/14-23-08.177/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1/0/8/0/1/1

I noticed this originally at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1778121070

will-moore commented 10 months ago

Another issue importing with this PR: Channels - see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1779595416 EDIT: This doesn't happen with OMEZarrReader-0.3.1-jar-with-dependencies.jar but does with OMEZarrReader_performance_allfiles.jar. https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1777374620

dgault commented 10 months ago

Updated the reader to allow setting of the list pixels flag via an environment variable (OME_ZARR_LIST_PIXELS).

Also added an additional method for reloading the options file. Though at the minute it seems that the cache directory which we had hoped to use is only available at the Memoizer level. So we will need to find a way to get that cache location into the reader

dgault commented 10 months ago

Updated with a commit that will force the reader to attempt to locate a bfoptions from the same location as the top level folder. Local testing shows this working even when the top level folder is a symlink. This will be the location it checks no matter which file within the dataset is initialized. If your top level folder is called myData.ome.zarr then you will need an options file beside it called myData.ome.zarr.bfoptions.

The options file itself should simply contain:

omezarr.list_pixels=false

will-moore commented 10 months ago

Testing today's build on idr-testing...

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/586/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p664_b586.jar
...etc, retstart...

Look for idr0091 images that don't already have memo file created - generates ResourceError: http://localhost:1080/webclient/?show=image-10648049

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zattrs

As omero-server...

echo "omezarr.list_pixels=false" > /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions

check...

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff
total 8.0K
drwxr-sr-x.  2 omero-server omero-server  114 Oct 31 10:15 .
drwxrwsr-x. 25 omero-server omero-server 4.0K Oct 11 09:49 ..
lrwxrwxrwx.  1 omero-server omero-server  109 Oct 11 09:46 d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr -> /bia-integrator-data/S-BIAD852/d28d77b1-5807-47fc-b3a6-57e58b93632f/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr
-rw-r--r--.  1 omero-server omero-server   26 Oct 31 10:15 d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions

When trying to view the image again, unfortunately I still see an error... Blitz logs...

2023-10-31 10:20:29,917 WARN  [              loci.formats.FormatHandler] (l.Server-9) Mismatch with group key paths and original OME-XML metadata, original ordering wont be maintained
2023-10-31 10:20:33,227 ERROR [              loci.formats.FormatHandler] (l.Server-9) ZarrReader loaded options file from: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions
2023-10-31 10:20:33,227 ERROR [              loci.formats.FormatHandler] (l.Server-9) ZarrReader getUsed files, skipPixels: false
2023-10-31 10:20:33,228 ERROR [              loci.formats.FormatHandler] (l.Server-9) ZarrReader fetching list of used files: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr
2023-10-31 10:20:33,228 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zattrs
2023-10-31 10:20:33,228 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zgroup
2023-10-31 10:20:33,229 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/0/.zarray
2023-10-31 10:20:33,231 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/0/0/0/0/0
...
...
58b93632f.zarr/3/98/2/0/0
2023-10-31 10:20:37,032 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/0/0/0
2023-10-31 10:20:37,032 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/1/0/0
2023-10-31 10:20:37,033 ERROR [              loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/2/0/0
2023-10-31 10:20:37,058 WARN  [   o.s.b.repo.ManagedReaderSecurityCheck] (l.Server-9) file demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/3/417/0/0/0 does not have a unique readab
le fileset
2023-10-31 10:20:37,058 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1698747619235] time[17823] tag[loci.formats.Memoizer.setId]
2023-10-31 10:20:37,059 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-9) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-4
7fc-b3a6-57e58b93632f.zarr/.zattrs
2023-10-31 10:20:37,060 ERROR [                ome.io.nio.PixelsService] (l.Server-9) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-
57e58b93632f.zarr/.zattrs
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/3/417/0/0/0 does not have a unique readable fileset
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor316.invoke(Unknown Source)

will-moore commented 10 months ago

With that last fix, updated ZarrReader and the Image is viewable! 👍

joshmoore commented 10 months ago

Woo hoo! 🎉

will-moore commented 10 months ago

Testing import on idr-testing with the bfoptions supported ZarrReader...

Import idr0054 sample image as above - regular import (including chunks) from s3...

omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_bfoptions.log --errs /tmp/idr0054_bfoptions.err

...
2023-11-01 15:49:26,804 343015     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314593
==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:05:29.496

will-moore commented 10 months ago

Updated the proxy idr-testing omeroreadonly-X servers with same OMEZarrReader_p665_b587.jar as tested above on idr-testing omeroreadwrite server.

will-moore commented 10 months ago

Testing whether that last commit fixes issue with mismatching pixel values on multi-well plates (that contain OME/METADATA.ome.xml) as reported https://github.com/IDR/idr-metadata/issues/675#issuecomment-1794822262 Updated ZarrReader on idr-testing to today's build jar named OMEZarrReader_p672_b594.jar

will-moore commented 10 months ago

The Channels issue above seems to be fixed 👍

However, ManagedRepo paths issue remains (see https://github.com/ome/ZarrReader/pull/64#issuecomment-1780650291) Importing idr0125 Plates e.g. original data is at /ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr on idr0125-pilot, I get paths like this (which shouldn't contain ngff/idr0125/):

demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-11/07/09-32-28.422/ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr/C/10/8/.zgroup

dgault commented 10 months ago

@will-moore, with that last commit I believe the file path issue should be resolved. It seems that getRequiredDirectories is used to trim the paths down to remove the common path, see https://github.com/ome/omero-blitz/blob/8d18dfcecc63b002ad89e924e6f03763b16052de/src/main/java/ome/services/blitz/repo/ManagedRepositoryI.java#L1467. For Zarr files we should only need one level below the common parent, that being the .zarr folder itself.