Closed dgault closed 9 months ago
Conflicting PR. Removed from build BIOFORMATS-push#94. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build BIOFORMATS-push#22. See the console output for more details. Possible conflicts:
--conflicts Conflict resolved in build BIOFORMATS-push#95. See the console output for more details.
Updated ZarrReader on idr-testing as described at https://github.com/IDR/idr-metadata/issues/672#issuecomment-1713858312 (after having used mkngff
on 3 Plates from idr0025).
Then tried viewing images from Plate 2 for the first time.
Logs say it took 95 mins for setid, but images appear black - see https://github.com/IDR/idr-metadata/issues/672#issuecomment-1714112834
Today's build took 4 mins for setId on idr0025 Plate 3: https://github.com/IDR/idr-metadata/issues/672#issuecomment-1715552599
Conflicting PR. Removed from build BIOFORMATS-push#99. See the console output for more details. Possible conflicts:
--conflicts Conflict resolved in build BIOFORMATS-push#100. See the console output for more details.
Testing "black" images OMEZarrReader using build from 11th Sept when I first noticed it (today's didn't include this PR)...
Testing on idr0125-pilot
...
ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1040:localhost:80
sudo -u omero-server -s
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/534/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
# as wmoore...
sudo service omero-server restart
Tried viewing a Plate P170
from idr0004 where I'd run mkngff and viewed the images using previous build of ZarrReader -
https://github.com/IDR/idr-metadata/issues/637#issuecomment-1708133932
On this plate, images rendered fine - looks good.
Then I tried viewing a different Plate where I'd run mkngff at the same time but not viewed the images (no memo file created). Plate P144
, e.g. http://localhost:1040/webclient/?show=image-696604
Memo file created just now...
$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "08-34-48.864_mkngff"
2023-09-13 09:35:24,553 DEBUG [ loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/..zattrs.bfmemo (219313 bytes)
2023-09-13 09:35:24,553 DEBUG [ loci.formats.Memoizer] (l.Server-7) start[1694597666657] time[57896] tag[loci.formats.Memoizer.setId]
2023-09-13 09:35:24,553 INFO [ ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr/.zattrs Series: 0
But this image is black, even with the rendering settings adjusted to be bright:
Also, the black images are rendering very slowly - ~25 seconds, compared to Plate P170
above where the images are rendering correctly at "normal" speeds (less than 0.5 secs).
Discussed this morning - improved logging, comparison between memo file generation on import and after mkngff.
Update today's build of ZarrReader on idr-testing:omeoreadwrite
...
Then tried viewing 3rd Plate from idr0025: http://localhost:1080/webclient/?show=image-3260835
3-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IContainer
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery(omero.api._IQueryTie@ab037ce6)
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IQuery
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin(omero.api._IAdminTie@8040fd4d)
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IAdmin
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript(omero.api._IScriptTie@b8b56855)
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IScript
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Unregistered servant:1a6468ae-7280-486a-98ea-2844e267cdd8/a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig(omero.api._IConfigTie@a6cf999d)
2023-09-15 09:41:18,674 INFO [ omero.cmd.SessionI] (l.Server-4) Removed servant from adapter: a8ff6aea-c650-4fbb-83e5-809ac842fc22omero.api.IConfig
2023-09-15 09:41:23,692 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:23,800 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/1
2023-09-15 09:41:23,995 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,106 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/2
2023-09-15 09:41:24,190 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/0
2023-09-15 09:41:24,300 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/0/3
2023-09-15 09:41:24,517 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,743 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 5 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/1
2023-09-15 09:41:24,803 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:24,955 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 6 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/2
2023-09-15 09:41:25,212 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 4 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/0
2023-09-15 09:41:25,368 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 7 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/1/3
2023-09-15 09:41:25,674 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:25,830 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 9 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/1
2023-09-15 09:41:25,899 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,006 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 10 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/2
2023-09-15 09:41:26,122 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 8 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/0
2023-09-15 09:41:26,232 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 11 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/2/3
2023-09-15 09:41:26,544 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,794 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 13 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/1
2023-09-15 09:41:26,863 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:26,975 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 14 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/2
2023-09-15 09:41:27,038 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 12 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/0
2023-09-15 09:41:27,113 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 15 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/1/3/3
2023-09-15 09:41:27,576 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,741 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 17 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/1
2023-09-15 09:41:27,814 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:27,925 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 18 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/2
2023-09-15 09:41:27,993 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 16 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/0
2023-09-15 09:41:28,099 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 19 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/0/3
2023-09-15 09:41:28,508 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,811 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 21 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/1
2023-09-15 09:41:28,874 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:28,983 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 22 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/2
2023-09-15 09:41:29,119 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 20 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/0
2023-09-15 09:41:29,229 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 23 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/1/3
2023-09-15 09:41:29,509 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,677 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 25 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/1
2023-09-15 09:41:29,753 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:29,868 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 26 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/2
2023-09-15 09:41:29,940 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 24 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/0
2023-09-15 09:41:30,101 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 27 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/2/3
2023-09-15 09:41:30,410 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,572 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 29 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/1
2023-09-15 09:41:30,637 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,706 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 30 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/2
2023-09-15 09:41:30,764 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 28 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/A/2/3/0
2023-09-15 09:41:30,971 INFO [ loci.formats.FormatHandler] (l.Server-9) Opening zarr for series 31 at path: /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e
...
23-09-15 09:50:33,759 INFO [ omero.cmd.SessionI] (l.Server-8) Unregistered servant:fa0664ae-6693-4e84-ad2a-c48335260413/472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer(omero.api._IContainerTie@be549585)
2023-09-15 09:50:33,759 INFO [ omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: 472dad67-a768-4f69-90b5-8837bc22336eomero.api.IContainer
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo9156189850653740893
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) start[1694771435991] time[104] tag[loci.formats.Memoizer.saveMemo]
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO [ ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
2023-09-15
First attempt to view the Image above gave this in webclient...
File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1008, in getRenderingDefId
return _M_omero.api.RenderingEngine._op_getRenderingDefId.invoke(self, ((), _ctx))
Ice.ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero
But on next attempt it worked and Image is NOT black!
$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "15-26-46.818_mkngff"
2023-09-12 11:08:39,631 DEBUG [ loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-12 11:08:39,631 DEBUG [ loci.formats.Memoizer] (l.Server-2) start[1694516674666] time[244964] tag[loci.formats.Memoizer.setId]
2023-09-12 11:08:39,631 INFO [ ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-13 23:00:16,310 DEBUG [ loci.formats.Memoizer] (.Server-13) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2018-01/24/14-20-58.331_mkngff/8c168760-7bc0-4692-90b1-c774711e7dd8.zarr/OME/.METADATA.ome.xml.bfmemo (666953 bytes)
--
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 09:50:36,096 DEBUG [ loci.formats.Memoizer] (l.Server-9) start[1694770724074] time[712022] tag[loci.formats.Memoizer.setId]
2023-09-15 09:50:36,097 INFO [ ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0
712022 ms is 12 minutes.
Previous plates on idr0125 have taken 95 minutes for Plate 1 and 4 minutes also on Plate 3 on idr-testing server.
Comparing regular import (no chunks) and timing of memo file creation with after mkngff...
On idr0125-pilot (we already have NGFF data without chunks) - update ZarrReader as above...
cd /ngff/idr0025
for dir in *; do
omero import --transfer=ln_s --depth=100 --name="${dir/.ome.zarr/}" --skip=all "$dir" --file "/tmp/$dir.log" --errs "/tmp/$dir.err";
done
But err logs have:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-15 10:39:17,672 294 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-15 10:39:17,691 313 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 995067a5752310c81d89accc34337b4b1d0624df date: 20230915-0010
2023-09-15 10:39:17,753 375 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-15 10:39:17,754 376 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-15 10:39:17,755 377 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-15 10:39:17,755 377 [ main] INFO formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-15 10:39:17,755 377 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-15 10:39:17,758 380 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-15 10:39:18,155 777 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-09-15 10:40:44,670 87292 [ main] INFO ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 78529ms. (86515ms total) [0 unknowns]
2023-09-15 10:40:44,722 87344 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-15 10:40:45,306 87928 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-15 10:40:45,960 88582 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-15 10:40:45,973 88595 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found
Reverted ZarrReader jar to OMEZarrReader-0.3.1-jar-with-dependencies.jar
on idr0125-pilot
and re-ran import of idr0025
plates above...
e.g.
2023-09-15 11:36:42,662 1321330 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287669
==> Summary
2509 files uploaded, 1 fileset, 1 plate created, 384 images imported, 0 errors in 0:21:56.370
Blank images (no chunks) are viewable.
I see several "saved memo" events in the logs. Times are 1.5 - 3.5 seconds.
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "11-14-46.693/10x"
2023-09-15 11:36:08,898 DEBUG [ loci.formats.Memoizer] (2-thread-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503832 bytes)
2023-09-15 11:36:08,899 DEBUG [ loci.formats.Memoizer] (2-thread-1) start[1694777765366] time[3532] tag[loci.formats.Memoizer.setId]
2023-09-15 11:36:09,000 INFO [ o.s.blitz.repo.ManagedImportRequestI] (2-thread-1) File format: Zarr
--
2023-09-15 12:02:14,851 DEBUG [ loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,851 DEBUG [ loci.formats.Memoizer] (l.Server-6) start[1694779333111] time[1740] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,852 INFO [ ome.io.nio.PixelsService] (l.Server-6) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 300
--
2023-09-15 12:02:14,899 DEBUG [ loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (1503823 bytes)
2023-09-15 12:02:14,899 DEBUG [ loci.formats.Memoizer] (l.Server-2) start[1694779333111] time[1787] tag[loci.formats.Memoizer.setId]
2023-09-15 12:02:14,899 INFO [ ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml Series: 132
Rendering of image planes in webclient takes 200-300 ms.
Now update symlink as omero-server user... using https://github.com/IDR/idr-metadata/issues/647#issuecomment-1697382761
cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
rm -rf "10x images plate 1.ome.zarr"
ln -s /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr "10x images plate 1.ome.zarr"
Image can be viewed as normal - "Save to All" generates thumbs for the Plate.
Then removed the memo file...
rm -rf "/data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/.METADATA.ome.xml.bfmemo"
Trying to view the Image in webclient:
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908
Will try mkngff
on idr0025/Plate 1
(with deleted memo file) from above...
(mkngff) bash-4.2$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287669 "/bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr" > "5287669.sql"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15 // 11-14-46.693 for fileset 5287669
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr -> /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr
$ psql -U omero -d idr -h $DBHOST -f 5287669.sql
UPDATE 384
BEGIN
mkngff_fileset
----------------
5287719
(1 row)
COMMIT
Viewing image...
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
}
Blitz.log
2023-09-15 13:31:52,866 INFO [ ome.security.basic.EventHandler] (l.Server-3) Auth: user=52,group=3,event=326848895(User),sess=6abc8eea-4910-4f1b-b57d-cf8b244189d8
2023-09-15 13:31:53,391 DEBUG [ loci.formats.Memoizer] (l.Server-8) start[1694784712636] time[755] tag[loci.formats.Memoizer.setId]
2023-09-15 13:31:53,391 ERROR [ ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
2023-09-15 13:31:53,391 ERROR [ ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
at jdk.internal.reflect.GeneratedMethodAccessor340.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)
Checked that we only have a single unique original file path/name:
idr=> select id from OriginalFile where path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-09/15/11-14-46.693_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/' and name = 'METADATA.ome.xml';
id
----------
52318508
(1 row)
EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908
Try on Plate 2
omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287670 "/bia-integrator-data/S-BIAD846/52304cdf-4eba-4f0a-84b1-690e0d66add9/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr" > 5287670.sql
(took 8 minutes)...
$ psql -U omero -d idr -h $DBHOST -f 5287670.sql
UPDATE 384
BEGIN
mkngff_fileset
----------------
5287720
(1 row)
COMMIT
Not working...
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-09/15/11-36-48.798_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml
}
EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908
mkngff
hasn't been working today on NGFF Filesets.
Going to try again on original .pattern
file data...
On idr0125-pilot... still trying with OMEZarrReader-0.3.1-jar-with-dependencies.jar
Try with some Plates from idr0011
...
Try with first 4 Plates from first screen...
idr0011.csv
idr0011/Plate1-Blue-A.ome.zarr,S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f,21689
idr0011/Plate1-Blue-B.ome.zarr,S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1,21687
idr0011/Plate1-Green-A.ome.zarr,S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b,21688
idr0011/Plate1-Green-B.ome.zarr,S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b,21685
$ for r in $(cat $IDRID.csv); do
> biapath=$(echo $r | cut -d',' -f2)
> uuid=$(echo $biapath | cut -d'/' -f2)
> fsid=$(echo $r | cut -d',' -f3)
> omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-56-03.328 for fileset 21689
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr -> /bia-integrator-data/S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-54-46.325 for fileset 21687
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-54-46.325_mkngff/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr -> /bia-integrator-data/S-BIAD866/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1/2e0f26b4-80a6-40d2-a054-2c7aee53dfd1.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-55-31.533 for fileset 21688
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-55-31.533_mkngff/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr -> /bia-integrator-data/S-BIAD866/c301bbe9-8e14-4760-bb1f-8081a130472b/c301bbe9-8e14-4760-bb1f-8081a130472b.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-43-55.929 for fileset 21685
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-43-55.929_mkngff/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr -> /bia-integrator-data/S-BIAD866/35d14872-e021-44fa-9c90-f0f1b55b165b/35d14872-e021-44fa-9c90-f0f1b55b165b.zarr
bash-4.2$ for r in $(cat $IDRID.csv); do
> fsid=$(echo $r | cut -d',' -f3)
> psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
> done
UPDATE 48
BEGIN
mkngff_fileset
----------------
5287721
(1 row)
COMMIT
UPDATE 48
BEGIN
mkngff_fileset
----------------
5287722
(1 row)
COMMIT
UPDATE 48
BEGIN
mkngff_fileset
----------------
5287723
(1 row)
COMMIT
UPDATE 46
BEGIN
mkngff_fileset
----------------
5287724
(1 row)
COMMIT
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/OME/METADATA.ome.xml
}
Something wrong with omero-mkngff
?
All above today was testing with https://github.com/IDR/omero-mkngff/pull/9 (which shouldn't behave any differently).
But lets revert to version I've been using mostly...
pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'
Try with 2 more plates from idr0011...
idr0011/Plate1-Red-A.ome.zarr,S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d,21692
idr0011/Plate1-Red-B.ome.zarr,S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086,21686
$ for r in $(cat $IDRID.csv); do biapath=$(echo $r | cut -d',' -f2); uuid=$(echo $biapath | cut -d'/' -f2); fsid=$(echo $r | cut -d',' -f3); omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"; done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 14-00-14.412 for fileset 21692
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr -> /bia-integrator-data/S-BIAD866/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 13-48-45.381 for fileset 21686
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-48-45.381_mkngff/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr -> /bia-integrator-data/S-BIAD866/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086/ebedb60b-cc7f-4c31-bdb0-9ca3cbcda086.zarr
$ for r in $(cat $IDRID.csv); do fsid=$(echo $r | cut -d',' -f3); psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"; done
UPDATE 48
BEGIN
mkngff_fileset
----------------
5287725
(1 row)
COMMIT
UPDATE 48
BEGIN
mkngff_fileset
----------------
5287726
(1 row)
COMMIT
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-00-14.412_mkngff/805d1ecc-8514-40ee-87bb-ba0c0a9fc60d.zarr/OME/METADATA.ome.xml
}
EDIT - Ignore this failure. See https://github.com/ome/ZarrReader/pull/64#issuecomment-1723143908
Looking at the Blitz log for the last exception above (viewing http://localhost:1080/webclient/?show=well-1227572 from idr0011/Plate1-Red-A.ome.zarr
I see...
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284)
at loci.formats.FormatReader.setId(FormatReader.java:1443)
at loci.formats.ImageReader.setId(ImageReader.java:849)
2023-09-18 07:58:56,503 INFO [ org.perf4j.TimingLogger] (l.Ser
Looking at the code... https://github.com/ome/bioformats/blob/bc1fcddd5329d900853d55c6f8378e9235a26a53/components/formats-api/src/loci/formats/FormatReader.java#L960
Here, core.size()
is returning 0
so that we get coreIndexToSeries(-1)
being called, which throws exception above.
Discussed Friday's testing at IDR meeting today - I realise that the released OMEZarrReader-0.3.1-jar-with-dependencies.jar
is likely failing for mkngff
workflow because it is missing one of the currently open PRs:
This one (#64), #58, #53 or #20.
So I will return to testing mkngff
workflow with the merge-ci build...
Using todays merge-ci build of ZarrReader: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/617/console
2023-09-18 00:06:16,745 [ scc.merge] INFO Repository: ome/ZarrReader
2023-09-18 00:06:16,745 [ scc.merge] INFO Excluded PRs:
2023-09-18 00:06:16,745 [ scc.merge] INFO - PR 58 dgault 'Add new option to list pixels in getUsedFiles' (exclude comment)
2023-09-18 00:06:16,745 [ scc.merge] INFO - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-09-18 00:06:16,745 [ scc.merge] INFO - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-09-18 00:06:16,745 [ scc.merge] INFO Already up-to-date.
2023-09-18 00:06:16,746 [ scc.merge] INFO
2023-09-18 00:06:16,746 [ scc.merge] INFO Merged PRs:
2023-09-18 00:06:16,746 [ scc.merge] INFO - PR 62 joshmoore 'Update README.md'
2023-09-18 00:06:16,746 [ scc.merge] INFO - PR 64 dgault 'ZarrReader performance improvements'
Testing memo file generation by viewing previously unviewed mkngff plate from idr0033:
Viewing image from 2nd plate: http://localhost:1080/webclient/?show=image-3230447 having checked the logs for memo file generation for that fileset to confirm that we haven't yet generated memo file..
grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.2 | grep -A 2 "2017-03/07/16-50-40.721_mkngff"
Checking on progress of memo file generation (started 3 hours ago)...
[wmoore@test120-omeroreadwrite ~]$ grep "Opening zarr for series" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2023-09-18 12:35:03,019 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,218 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/1
2023-09-18 12:35:03,289 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,392 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/2
2023-09-18 12:35:03,460 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/0
2023-09-18 12:35:03,573 INFO [ loci.formats.FormatHandler] (l.Server-3) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/A/1/0/3
2023-09-18 12:44:28,495 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,572 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,691 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:44:28,748 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,816 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:44:28,818 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,860 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:44:28,921 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:44:28,922 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:28,924 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:44:29,026 INFO [ loci.formats.FormatHandler] (l.Server-2) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:44:29,027 INFO [ loci.formats.FormatHandler] (l.Server-0) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:52:38,383 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,550 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:52:38,615 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,733 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:52:38,781 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:52:38,896 INFO [ loci.formats.FormatHandler] (l.Server-4) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:56:49,997 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,012 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,154 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:56:50,154 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 12:56:50,202 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,202 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,278 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:56:50,303 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 12:56:50,325 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,326 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 12:56:50,400 INFO [ loci.formats.FormatHandler] (l.Server-5) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 12:56:50,440 INFO [ loci.formats.FormatHandler] (l.Server-7) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:40:54,300 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,415 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:40:54,463 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,532 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:40:54,579 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:40:54,683 INFO [ loci.formats.FormatHandler] (.Server-11) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:56:45,188 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,188 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,629 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:56:45,638 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 13:56:45,709 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,716 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,849 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:56:45,851 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 13:56:45,914 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,914 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 13:56:45,979 INFO [ loci.formats.FormatHandler] (.Server-10) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 13:56:46,042 INFO [ loci.formats.FormatHandler] (.Server-13) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
2023-09-18 14:43:00,991 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,140 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 1 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/1
2023-09-18 14:43:01,203 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,308 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 2 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/2
2023-09-18 14:43:01,436 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 0 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/0
2023-09-18 14:43:01,551 INFO [ loci.formats.FormatHandler] (l.Server-6) Opening zarr for series 3 at path: /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/A/1/0/3
All these logs are from the first field of A/1
at A/1/0
.
Attempting to view a different Well or Field from that Plate doesn't show any more similar logs. Presumably some flag indicates that memo file generation is in progress already...?
Images from Plate 2 are finally visible today. Strange in logs below that we have logs from several "saved memo" events overnight: NB: add 1 hour for BST: 23:04:42, 01:52:22, 02:49:16, 04:15:45, 04:15:46, 38519781 ms is 10.6 hours. 47240506 ms is 13 hours
grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "16-50-40.721_mkngff/e45c"
2023-09-18 23:04:42,825 DEBUG [ loci.formats.Memoizer] (l.Server-0) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,825 DEBUG [ loci.formats.Memoizer] (l.Server-0) start[1695039763044] time[38519781] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,826 INFO [ ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-18 23:04:42,841 DEBUG [ loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,841 DEBUG [ loci.formats.Memoizer] (l.Server-2) start[1695040253748] time[38029092] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,841 INFO [ ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-18 23:04:42,973 DEBUG [ loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-18 23:04:42,973 DEBUG [ loci.formats.Memoizer] (l.Server-4) start[1695040759427] time[37523545] tag[loci.formats.Memoizer.setId]
2023-09-18 23:04:42,973 INFO [ ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,626 DEBUG [ loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,626 DEBUG [ loci.formats.Memoizer] (l.Server-7) start[1695041102119] time[47240506] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,626 INFO [ ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,822 DEBUG [ loci.formats.Memoizer] (.Server-11) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,822 DEBUG [ loci.formats.Memoizer] (.Server-11) start[1695043675269] time[44667552] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,822 INFO [ ome.io.nio.PixelsService] (.Server-11) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 01:52:22,856 DEBUG [ loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 01:52:22,856 DEBUG [ loci.formats.Memoizer] (l.Server-5) start[1695041096479] time[47246376] tag[loci.formats.Memoizer.setId]
2023-09-19 01:52:22,856 INFO [ ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 02:49:16,679 DEBUG [ loci.formats.Memoizer] (.Server-10) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,679 DEBUG [ loci.formats.Memoizer] (.Server-10) start[1695044671046] time[47085633] tag[loci.formats.Memoizer.setId]
2023-09-19 02:49:16,680 INFO [ ome.io.nio.PixelsService] (.Server-10) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 02:49:16,950 DEBUG [ loci.formats.Memoizer] (.Server-13) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,950 DEBUG [ loci.formats.Memoizer] (.Server-13) start[1695044686175] time[47070775] tag[loci.formats.Memoizer.setId]
2023-09-19 02:49:16,950 INFO [ ome.io.nio.PixelsService] (.Server-13) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 02:49:16,958 DEBUG [ loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 02:49:16,958 INFO [ org.perf4j.TimingLogger] (2-thread-2) start[1695091756958] time[0] tag[omero.call.success.ome.services.RenderingBean.close]
2023-09-19 02:49:16,959 INFO [ ome.services.util.ServiceHandler] (2-thread-2) Rslt: null
--
2023-09-19 04:15:45,698 DEBUG [ loci.formats.Memoizer] (.Server-14) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,699 DEBUG [ loci.formats.Memoizer] (.Server-14) start[1695066384572] time[30561126] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,699 INFO [ ome.io.nio.PixelsService] (.Server-14) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,837 DEBUG [ loci.formats.Memoizer] (.Server-20) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,837 DEBUG [ loci.formats.Memoizer] (.Server-20) start[1695058260700] time[38685136] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,837 INFO [ ome.io.nio.PixelsService] (.Server-20) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,935 DEBUG [ loci.formats.Memoizer] (l.Server-8) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,935 DEBUG [ loci.formats.Memoizer] (l.Server-8) start[1695051155880] time[45790055] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,936 INFO [ ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:45,949 DEBUG [ loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:45,949 DEBUG [ loci.formats.Memoizer] (l.Server-1) start[1695050732463] time[46213485] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:45,949 INFO [ ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,090 DEBUG [ loci.formats.Memoizer] (.Server-12) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,090 DEBUG [ loci.formats.Memoizer] (.Server-12) start[1695051103810] time[45842280] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,090 INFO [ ome.io.nio.PixelsService] (.Server-12) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,210 DEBUG [ loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,210 DEBUG [ loci.formats.Memoizer] (l.Server-3) start[1695051108480] time[45837730] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,210 INFO [ ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,253 DEBUG [ loci.formats.Memoizer] (.Server-16) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,253 DEBUG [ loci.formats.Memoizer] (.Server-16) start[1695050749658] time[46196595] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,253 INFO [ ome.io.nio.PixelsService] (.Server-16) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-19 04:15:46,377 DEBUG [ loci.formats.Memoizer] (.Server-15) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/.METADATA.ome.xml.bfmemo (6667235 bytes)
2023-09-19 04:15:46,377 DEBUG [ loci.formats.Memoizer] (.Server-15) start[1695051598118] time[45348259] tag[loci.formats.Memoizer.setId]
2023-09-19 04:15:46,377 INFO [ ome.io.nio.PixelsService] (.Server-15) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr/OME/METADATA.ome.xml Series: 9
As discussed after standup this morning:
2023-09-15 10:39:17,758 380 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
With OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
from David with ERROR logging, updated onto idr-testing, restart etc.
On idr-testing, as omero-server user, made a local copy of the first Plate from idr0011 for testing...
$ pwd
/opt/omero/server/ngff/idr0011
$ cp -R /bia-integrator-data/S-BIAD866/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr .
Did regular import of Plates with chunks:omero import --depth=10 f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
Log file lists every image etc for each chunk. E.g. .zarr/A/1/0/0/0/1/7/0/0
:
...
2023-09-19 10:33:16,010 309337 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/7/0/2
2023-09-19 10:33:16,010 309337 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/0
2023-09-19 10:33:16,010 309337 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 10:33:16,010 309337 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 10:33:16,010 309337 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 10:33:16,011 309338 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 10:33:16,013 309340 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 10:33:16,412 309739 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/A/1/0/0/0/1/7/0/0
2023-09-19 10:33:16,412 309739 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 10:33:16,412 309739 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
...
2023-09-19 11:50:42,622 4955949 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 11:50:42,622 4955949 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 11:50:42,622 4955949 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 11:50:42,622 4955949 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 11:50:42,623 4955950 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 11:50:42,987 4956314 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/A/4/0/1/0/1/13/0/0
2023-09-19 11:50:42,987 4956314 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 11:50:42,987 4956314 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
...
2023-09-19 20:11:58,424 35031751 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/0
2023-09-19 20:11:58,424 35031751 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/1
2023-09-19 20:11:58,424 35031751 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : F/8/0/2
2023-09-19 20:11:58,424 35031751 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 20:11:58,424 35031751 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 20:11:58,425 35031752 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 20:11:58,790 35032117 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/C/8/0/2/0/2/3/0/0
2023-09-19 20:11:58,790 35032117 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /opt/omero/server/ngff/idr0011/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr
2023-09-19 20:11:58,790 35032117 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
Cancelled after 20 hours, processing 24th image out of 48.
Try to retest the https://github.com/ome/ZarrReader/pull/64#issuecomment-1721072253 "No imports found" above.
Updated today's OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
(with ERROR logging) onto idr0125-pilot...
Then
$ cd /ngff/idr0025
$ omero import --transfer=ln_s --skip=all --depth=10 "10x images plate 1.ome.zarr" --file /tmp/idr0025_20230919.log --err /tmp/idr0025_20230919.err
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-19 12:52:15,840 306 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-19 12:52:15,860 326 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 6371c790a70306bbfcf1365d64bad04c19fc538d date: 2023-09-19T09:40:05Z
2023-09-19 12:52:15,928 394 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-19 12:52:15,929 395 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-19 12:52:15,929 395 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-19 12:52:15,930 396 [ main] INFO formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-19 12:52:15,930 396 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-19 12:52:15,933 399 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-19 12:52:16,337 803 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-09-19 12:52:16,453 919 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-19 12:52:16,460 926 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-19 12:52:16,468 934 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-09-19 12:52:17,336 1802 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-19 12:52:17,353 1819 [ main] ERROR loci.formats.FormatHandler - ZarrReader generating group keys
2023-09-19 12:52:17,353 1819 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-09-19 12:52:46,916 31382 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-09-19 12:52:46,920 31386 [ main] ERROR loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-09-19 12:52:46,960 31426 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-19 12:52:46,960 31426 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:46,981 31447 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:46,982 31448 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-19 12:52:46,982 31448 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1
2023-09-19 12:52:46,990 31456 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:46,991 31457 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-19 12:52:46,991 31457 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:46,991 31457 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2
2023-09-19 12:52:47,006 31472 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:47,007 31473 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3
2023-09-19 12:52:47,007 31473 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-19 12:52:47,008 31474 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/3
2023-09-19 12:52:47,022 31488 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-19 12:52:47,022 31488 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-19 12:52:47,022 31488 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-19 12:52:47,022 31488 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-19 12:52:47,022 31488 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3
...
2023-09-19 12:52:47,093 31559 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-19 12:52:47,093 31559 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-19 12:52:47,093 31559 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-19 12:52:47,093 31559 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-19 12:52:47,139 31605 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-19 12:52:47,170 31636 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-19 12:52:47,320 31786 [ main] INFO ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 30872ms. (30983ms total) [0 unknowns]
2023-09-19 12:52:47,367 31833 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-19 12:52:47,872 32338 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-19 12:52:48,825 33291 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-19 12:52:48,844 33310 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-19 12:52:48,845 33311 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-19 12:52:48,845 33311 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-19 12:52:48,845 33311 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-19 12:52:48,845 33311 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-19 12:52:48,845 33311 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found
Testing performance of memo file generation on Plates from idr0011 on idr-testing...
http://localhost:1080/webclient/?show=well-1227779
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "888_mkngff/5fbc36b"
2023-09-20 09:23:46,281 DEBUG [ loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo (6990617 bytes)
2023-09-20 09:23:46,281 DEBUG [ loci.formats.Memoizer] (l.Server-5) start[1695201778449] time[47832] tag[loci.formats.Memoizer.setId]
2023-09-20 09:23:46,282 INFO [ ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/METADATA.ome.xml Series: 1
47832 is 48 seconds
Deleted memo file, viewed image again and it got recreated
[wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo
logs
2023-09-20 09:31:01,247 DEBUG [ loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/.METADATA.ome.xml.bfmemo (6990617 bytes)
2023-09-20 09:31:01,247 DEBUG [ loci.formats.Memoizer] (l.Server-2) start[1695202188874] time[72373] tag[loci.formats.Memoizer.setId]
2023-09-20 09:31:01,247 INFO [ ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-07-57.888_mkngff/5fbc36b9-ce8f-48a0-830a-ba1feabbb604.zarr/OME/METADATA.ome.xml Series: 1
72373 is 72 seconds (bit slower)?
Trying with next plate http://localhost:1080/webclient/?show=image-2850468 - creating memo, deleting then creating again gives similar times for both: 49 secs and 52 secs:
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "539_mkngff/196473f"
2023-09-20 09:33:49,246 DEBUG [ loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/.METADATA.ome.xml.bfmemo (6987787 bytes)
2023-09-20 09:33:49,246 DEBUG [ loci.formats.Memoizer] (l.Server-3) start[1695202379756] time[49490] tag[loci.formats.Memoizer.setId]
2023-09-20 09:33:49,246 INFO [ ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-20 09:35:33,753 DEBUG [ loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/.METADATA.ome.xml.bfmemo (6987787 bytes)
2023-09-20 09:35:33,753 DEBUG [ loci.formats.Memoizer] (l.Server-3) start[1695202481531] time[52222] tag[loci.formats.Memoizer.setId]
2023-09-20 09:35:33,753 INFO [ ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/14-12-06.539_mkngff/196473fb-348a-4cae-8c65-11a1dc996796.zarr/OME/METADATA.ome.xml Series: 0
Viewing plate 1 from idr0025 - triggered a memo file generation (without deleting previous one)...
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 ".590_mkngff/52304cd"
2023-09-20 09:51:18,564 DEBUG [ loci.formats.Memoizer] (l.Server-0) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/.METADATA.ome.xml.bfmemo (876328 bytes)
2023-09-20 09:51:18,564 DEBUG [ loci.formats.Memoizer] (l.Server-0) start[1695203292938] time[185625] tag[loci.formats.Memoizer.setId]
2023-09-20 09:51:18,564 INFO [ ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml Series: 0
185625 ms is 3 minutes
Plate 2 memo file already exists -rw-rw-r--. 1 omero-server omero-server 551K Sep 15 10:13 .METADATA.ome.xml.bfmemo
Previous time took 11 mins to generate:
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.1 | grep -A 2 "4.747_mkngff/72cc29"
2023-09-15 10:13:12,445 DEBUG [ loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (563699 bytes)
2023-09-15 10:13:12,445 DEBUG [ loci.formats.Memoizer] (l.Server-3) start[1694772108579] time[683865] tag[loci.formats.Memoizer.setId]
2023-09-15 10:13:12,445 INFO [ ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 76
Deleted...
[wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "4.747_mkngff/72cc29"
2023-09-20 10:02:39,729 DEBUG [ loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (876328 bytes)
2023-09-20 10:02:39,729 DEBUG [ loci.formats.Memoizer] (l.Server-4) start[1695203993000] time[166728] tag[loci.formats.Memoizer.setId]
2023-09-20 10:02:39,729 INFO [ ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 0
166728 is 2.7 minutes 👍
Looks like ZarrReader is assuming every Well is filled - getting exceptions from Plates with missing Wells: https://github.com/IDR/idr-metadata/issues/644#issuecomment-1727543657
Updated idr0125-pilot... (don't want to restart idr-testing server during long-running omero mkngff sql
generation)...
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/545/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
cp OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
...
sudo service omero-server restart
Test viewing of sparse Plate in idr0004 (previously generated):
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "185_mkngff/103d9428" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 10:34:53,599 DEBUG [ loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/..zattrs.bfmemo (171808 bytes)
2023-09-21 10:34:53,599 DEBUG [ loci.formats.Memoizer] (l.Server-7) start[1695292350587] time[143012] tag[loci.formats.Memoizer.setId]
2023-09-21 10:34:53,599 INFO [ ome.io.nio.PixelsService] (l.Server-7) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/.zattrs Series: 0
143012 ms is 2.4 minutes
Tried import again on idr0125-pilot...
cd /ngff/idr0025
omero import --transfer=ln_s --skip=all --depth=10 "10x images plate 1.ome.zarr" --file /tmp/idr0025_20230921.log --err /tmp/idr0025_20230921.err
err
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-09-21 12:14:58,287 286 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-09-21 12:14:58,308 307 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: 237c1009f320c633bbdec5b0937647a320c5c250 date: 20230921-0010
2023-09-21 12:14:58,384 383 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-09-21 12:14:58,386 385 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-09-21 12:14:58,386 385 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-09-21 12:14:58,386 385 [ main] INFO formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-09-21 12:14:58,387 386 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-09-21 12:14:58,392 391 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-09-21 12:14:58,772 771 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-09-21 12:14:58,849 848 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-21 12:14:58,855 854 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:58,862 861 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-09-21 12:14:59,533 1532 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:59,549 1548 [ main] ERROR loci.formats.FormatHandler - ZarrReader generating group keys
2023-09-21 12:14:59,556 1555 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-09-21 12:14:59,728 1727 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-09-21 12:14:59,736 1735 [ main] ERROR loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-09-21 12:14:59,761 1760 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-21 12:14:59,761 1760 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,775 1774 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,775 1774 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-21 12:14:59,775 1774 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1
2023-09-21 12:14:59,776 1775 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,776 1775 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
2023-09-21 12:14:59,776 1775 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,776 1775 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/3
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/3
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-09-21 12:14:59,777 1776 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/0
2023-09-21 12:14:59,778 1777 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/1
2023-09-21 12:14:59,778 1777 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/0/2
...
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/1
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/2
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/0/3
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/0
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/1
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/2
2023-09-21 12:14:59,817 1816 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : A/1/1/3
...
2023-09-21 12:14:59,839 1838 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/0
2023-09-21 12:14:59,839 1838 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-21 12:14:59,839 1838 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-21 12:14:59,840 1839 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-21 12:14:59,840 1839 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-21 12:14:59,888 1887 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-21 12:14:59,917 1916 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-21 12:14:59,938 1937 [ main] ERROR loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-09-21 12:14:59,939 1938 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-21 12:14:59,944 1943 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/0/.zarray
2023-09-21 12:14:59,944 1943 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/1/.zarray
2023-09-21 12:14:59,944 1943 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/A/1/0/2/.zarray
...
2023-09-21 12:15:00,031 2030 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/H/.zgroup
2023-09-21 12:15:00,031 2030 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/.zattrs
2023-09-21 12:15:00,031 2030 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/.zgroup
2023-09-21 12:15:00,031 2030 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0025/10x images plate 1.ome.zarr/OME/METADATA.ome.xml
2023-09-21 12:15:00,031 2030 [ main] ERROR loci.formats.FormatHandler - ZarrReader returning list of used files of size: 972
2023-09-21 12:15:00,104 2103 [ main] INFO ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 0 group(s) with 1 call(s) to setId in 1213ms. (1333ms total) [0 unknowns]
2023-09-21 12:15:00,149 2148 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-21 12:15:00,616 2615 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-21 12:15:01,070 3069 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-21 12:15:01,084 3083 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
No imports found
Repeating import with today's merge build of ZarrReader on idr0125-pilot...
repeating https://github.com/ome/ZarrReader/pull/64#issuecomment-1721072253
Import of idr0025 Plate 1 (without chunks): looking good...
2023-09-25 14:57:57,059 40298 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/0
2023-09-25 14:57:57,059 40298 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/1
2023-09-25 14:57:57,059 40298 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/2
2023-09-25 14:57:57,059 40298 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : H/12/3/3
2023-09-25 14:57:57,059 40298 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-09-25 14:57:57,109 40348 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-09-25 14:57:57,138 40377 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-09-25 14:57:57,156 40395 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0025/10x images plate 1.ome.zarr
2023-09-25 14:57:57,215 40454 [ main] ERROR loci.formats.FormatHandler - ZarrReader returning list of used files of size: 2509
2023-09-25 14:58:00,001 43240 [ main] INFO ome.formats.importer.ImportCandidates - 2509 file(s) parsed into 1 group(s) with 1 call(s) to setId in 27753ms. (41061ms total) [0 unknowns]
2023-09-25 14:58:01,603 44842 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-09-25 14:58:05,203 48442 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-09-25 14:58:06,700 49939 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-09-25 14:58:06,721 49960 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-09-25 14:58:06,721 49960 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-09-25 14:58:06,722 49961 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-09-25 14:58:06,722 49961 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-25 14:58:06,722 49961 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-25 14:58:06,722 49961 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-09-25 14:58:06,892 50131 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
...
Image:14993619
Image:14993620
Image:14993621
Image:14993622
==> Summary
2509 files uploaded, 1 fileset, 1 plate created, 672 images imported, 0 errors in 0:24:21.111
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/memory-error-on-zarr-import-on-omero/86684/2
Conflicting PR. Removed from build BIOFORMATS-push#635. See the console output for more details. Possible conflicts:
--conflicts
Conflicting PR. Removed from build BIOFORMATS-push#636. See the console output for more details. Possible conflicts:
--conflicts Conflict resolved in build BIOFORMATS-push#637. See the console output for more details.
As discussed this morning, those getUsedFiles
changes fix the errors reported at https://github.com/IDR/idr-utils/pull/56#issuecomment-1746821150. However, it is also expected that they will break the regular import of NGFF data (ignoring chunks).
Want to test whether the released ZarrReader (without this PR or getUsedFiles
changes) works when we import an image without chunks.
Updated idr-testing:omeroreadwrite
server with released jar... and restarted
bash-4.2$ cp OMEZarrReader-0.3.1-jar-with-dependencies.jar /opt/omero/server/OMERO.server/lib/server/
bash-4.2$ cp OMEZarrReader-0.3.1-jar-with-dependencies.jar /opt/omero/server/OMERO.server/lib/client/
Download metadata only image from https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120.html
aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr .
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/.zattrs to ./.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/.zgroup to 0/.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/.zgroup to ./.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/.zattrs to 0/.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/0/.zarray to 0/0/.zarray
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/0/1/.zarray to 0/1/.zarray
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/.zgroup to OME/.zgroup
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/.zattrs to OME/.zattrs
download: s3://bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr/OME/METADATA.ome.xml to OME/METADATA.ome.xml
Import into idr-testing...
$ (venv3) [wmoore@test120-omeroreadwrite idr0051]$ omero import --depth=10 -d 3357 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr
Downloading https://downloads.openmicroscopy.org/omero/latest/OMERO.java.zip
Using OMERO.java-5.6.9-ice36
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Oct 05, 2023 11:21:28 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
2023-10-05 11:21:28,841 374 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2023-10-05 11:21:28,859 392 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2023-10-05 11:21:28,914 447 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-05 11:21:29,257 790 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-10-05 11:21:29,716 1249 [ main] INFO ome.formats.importer.ImportCandidates - 9 file(s) parsed into 1 group(s) with 1 call(s) to setId in 455ms. (459ms total) [0 unknowns]
2023-10-05 11:21:29,758 1291 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-05 11:21:30,259 1792 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-05 11:21:30,956 2489 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-05 11:21:30,978 2511 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.95.1.el7.x86_64
2023-10-05 11:21:31,604 3137 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Dataset:3357
2023-10-05 11:21:31,647 3180 [2-thread-1] INFO ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:3357
2023-10-05 11:21:31,651 3184 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-10-05 11:21:51,999 23532 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-10-05 11:21:52,021 23554 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml...
2023-10-05 11:21:52,120 23653 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
2023-10-05 11:21:52,225 23758 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
2023-10-05 11:21:52,270 23803 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs...
2023-10-05 11:21:52,335 23868 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs
2023-10-05 11:21:52,486 24019 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zattrs
2023-10-05 11:21:52,511 24044 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup...
2023-10-05 11:21:52,562 24095 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup
2023-10-05 11:21:52,696 24229 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/.zgroup
2023-10-05 11:21:52,727 24260 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs...
2023-10-05 11:21:52,782 24315 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs
2023-10-05 11:21:52,910 24443 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zattrs
2023-10-05 11:21:52,937 24470 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup...
2023-10-05 11:21:52,995 24528 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup
2023-10-05 11:21:53,053 24586 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/.zgroup
2023-10-05 11:21:53,139 24672 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray...
2023-10-05 11:21:53,189 24722 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray
2023-10-05 11:21:53,262 24795 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/0/.zarray
2023-10-05 11:21:53,291 24824 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray...
2023-10-05 11:21:53,348 24881 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray
2023-10-05 11:21:53,519 25052 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/0/1/.zarray
2023-10-05 11:21:53,548 25081 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs...
2023-10-05 11:21:53,599 25132 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs
2023-10-05 11:21:53,861 25394 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zattrs
2023-10-05 11:21:53,889 25422 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup...
2023-10-05 11:21:53,937 25470 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup
2023-10-05 11:21:54,002 25535 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/.zgroup
2023-10-05 11:21:54,810 26343 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-05 11:21:55,105 26638 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 51578493
2023-10-05 11:22:02,441 33974 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 51578493
2023-10-05 11:22:49,139 80672 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 51578493
2023-10-05 11:22:50,921 82454 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 51578493
2023-10-05 11:22:51,101 82634 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 51578493
2023-10-05 11:22:51,149 82682 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 51578493
2023-10-05 11:22:51,312 82845 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/OME/METADATA.ome.xml
Image:14874782
Other imported objects:
Fileset:6312209
==> Summary
9 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:01:20.030
This image shows up as black as expected (no chunks) but is viewable.
But also, the mkngff
version of the same image (https://github.com/IDR/idr-utils/pull/56#issuecomment-1746831543) is also still visible using OMEZarrReader-0.3.1-jar-with-dependencies.jar
, without getUsedFile
changes.
Strangely, the mkngff
image from idr0051 above that is viewable with OMEZarrReader-0.3.1-jar-with-dependencies.jar
appears to be the exception. Other mkngff Images and Plates are not working with that jar.
2023-10-05 11:48:05,961 INFO [ loci.formats.ImageReader] (l.Server-0) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zar
r/OME/METADATA.ome.xml
2023-10-05 11:48:07,028 DEBUG [ loci.formats.Memoizer] (l.Server-0) start[1696506484058] time[2970] tag[loci.formats.Memoizer.setId]
2023-10-05 11:48:07,029 ERROR [ ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b1
14-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-05 11:48:07,030 ERROR [ ome.io.nio.PixelsService] (l.Server-0) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697
900e39.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
Try to import the original copy of the idr0051 image from s3 with chunks...
$ omero import --depth=10 -d 3357 /bia-integrator-data/S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120/c49efcfd-e767-4ae5-adbf-299cafd92120.zarr
...
...taking a long time to walk all chunks on s3....
Still with released ZarrReader 3.1, lets try mkngff
without Directories or chunks on a single Plate 1-23
from idr0010 (still unchanged on idr-testing).
$ psql -U omero -d idr -h $DBHOST -f 20751.sql
UPDATE 384
BEGIN
mkngff_fileset
----------------
6312259
(1 row)
COMMIT
(venv3) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 20751 "/bia-integrator-data/S-BIAD885/0046b0d0-f20b-4482-84b1-4b2b154865fd/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr -> /bia-integrator-data/S-BIAD885/0046b0d0-f20b-4482-84b1-4b2b154865fd/0046b0d0-f20b-4482-84b1-4b2b154865fd.zarr
http://localhost:1080/webclient/?show=well-1033384 ...
This gives a ResourceError
.
Updating to OMEZarrReader_b558_p635.jar
from https://github.com/IDR/idr-utils/pull/56#issuecomment-1746821150 fixes viewing of idr0010 plate:
[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "102_mkngff/e7bdccbf" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-10-05 15:01:49,707 DEBUG [ loci.formats.Memoizer] (l.Server-6) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (2316371 bytes)
2023-10-05 15:01:49,707 DEBUG [ loci.formats.Memoizer] (l.Server-6) start[1696517474813] time[634894] tag[loci.formats.Memoizer.setId]
2023-10-05 15:01:49,707 ERROR [ loci.formats.FormatHandler] (l.Server-1) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/A/9/0/2/0/1/0/0/0
--
2023-10-05 15:01:52,200 DEBUG [ loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (2316371 bytes)
2023-10-05 15:01:52,200 DEBUG [ loci.formats.Memoizer] (l.Server-1) start[1696517913517] time[198683] tag[loci.formats.Memoizer.setId]
2023-10-05 15:01:52,200 INFO [ ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 0
634894 ms is 10 minutes 198683 ms is 3 minutes
Testing different builds from jars build by @dgault... Copying jars and renaming...
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/main-canonicalPR-getUsedFilesAllFiles/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_canonical_allfiles.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/main-canonicalPR-getUsedFilesNoPixels/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_canonical_nopixels.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/PerformancePR-getUsedFilesNoPixels/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_performance_nopixels.jar
rsync -rvP idr0-slot3.openmicroscopy.org:/uod/idr/scratch/inbox/ZarrReaderTesting/PerformancePR-getUsedFilesAllFiles/OMEZarrReader-jar-with-dependencies.jar ./OMEZarrReader_performance_allfiles.jar
Upload each to idr-testing:omeroreadwrite server.
Copy them (as omero-server) to /opt/omero/server/OMERO.server/lib/server/
(and lib/server
) and restart OMERO.
Test viewing of mkngff filesets - if previously viewed, this was using the ZarrReader version described at https://github.com/IDR/idr-metadata/issues/675#issuecomment-1755198551
With OMEZarrReader_canonical_nopixels.jar
viewing previously viewed idr0054 images works. Viewing previously viewed idr0004 mkngff plate - I see lots of these logs in Blitz logs... This will take a long time to complete the plate...
i._IConfigTie@64645b60)
2023-10-18 11:48:25,453 INFO [ omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IConfig
2023-10-18 11:48:25,453 INFO [ omero.cmd.SessionI] (l.Server-8) Unregistered servant:8d74e6f3-5b9d-41ee-ac72-eb4839cc6b5e/e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IScript(omero.api._IScriptTie@7a387856)
2023-10-18 11:48:25,453 INFO [ omero.cmd.SessionI] (l.Server-8) Removed servant from adapter: e9e1176c-4599-494d-b0c0-da41e087184eomero.api.IScript
2023-10-18 11:48:25,499 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/4/0/0
2023-10-18 11:48:25,748 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/5/0/0
2023-10-18 11:48:25,962 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/6/0/0
2023-10-18 11:48:26,222 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/7/0/0
2023-10-18 11:48:26,443 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/8/0/0
2023-10-18 11:48:26,663 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/0/9/0/0
2023-10-18 11:48:26,991 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/0/0/0
2023-10-18 11:48:27,239 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/1/0/0
2023-10-18 11:48:27,587 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/2/0/0
2023-10-18 11:48:27,999 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/3/0/0
2023-10-18 11:48:28,370 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/4/0/0
2023-10-18 11:48:28,599 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/5/0/0
2023-10-18 11:48:28,836 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/6/0/0
2023-10-18 11:48:29,052 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/7/0/0
2023-10-18 11:48:29,311 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/8/0/0
2023-10-18 11:48:29,636 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/1/1/9/0/0
2023-10-18 11:48:29,745 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/.zarray
2023-10-18 11:48:30,158 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/0/0/0/0
2023-10-18 11:48:30,501 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-25-30.185_mkngff/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/7/0/2/0/1/0/0
2023-10-18 11:48:30,851 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07
Takes a long time, but eventually this plate is viewable.
Import as wmoore user (not inplace) from idr0054 on s3...
$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_nopixels.log --errs /tmp/idr0054_nopixels.err
No import found
Made metadata-only copy of image from idr0054 by copying image above, then deleting the chunks dir from each resolution.
Also gave No import found
.
Testing next with OMEZarrReader_canonical_allfiles.jar
...
$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_allpixels.log --errs /tmp/idr0054_allpixels.err
13:26:22,957 488833 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup...
2023-10-18 13:26:23,003 488879 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:26:23,176 489052 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:26:23,205 489081 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml...
2023-10-18 13:26:23,258 489134 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:26:23,347 489223 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:26:30,470 496346 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 13:26:30,830 496706 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61515641
2023-10-18 13:26:36,885 502761 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 61515641
2023-10-18 13:26:43,473 509349 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 61515641
2023-10-18 13:26:47,910 513786 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 61515641
2023-10-18 13:26:47,958 513834 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 61515641
2023-10-18 13:26:47,998 513874 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61515641
2023-10-18 13:26:48,201 514077 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314491
==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:08:20.580
Import of metadata-only image from idr0054.
$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_allfiles_mo.log --errs /tmp/idr0054_allfiles_mo.err
...
2023-10-18 13:29:15,112 7278 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:29:15,236 7402 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:29:15,541 7707 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 13:29:15,637 7803 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61516095
2023-10-18 13:29:16,216 8382 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 61516095
2023-10-18 13:29:21,080 13246 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 61516095
2023-10-18 13:29:21,636 13802 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 61516095
2023-10-18 13:29:21,691 13857 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 61516095
2023-10-18 13:29:21,729 13895 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61516095
2023-10-18 13:29:21,926 14092 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314492
==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:11.501
Black image but viewing works without errors.
Testing OMEZarrReader_performance_nopixels.jar
...
(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_allfiles.log --errs /tmp/idr0054_perf_allfiles.err
Import log repeats this a couple of times per second with all files being listed... Feels like infinite loop!
...
2023-10-18 13:50:33,051 119282 [ main] ERROR loci.formats.FormatHandler - ZarrReader including performance PR and getUsedFiles changes (skips pixels)
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/.zattrs
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/.zgroup
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/.zattrs
2023-10-18 13:50:33,297 119528 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/.zgroup
2023-10-18 13:50:33,298 119529 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/.zarray
2023-10-18 13:50:33,298 119529 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
2023-10-18 13:50:33,298 119529 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/1
2023-10-18 13:50:33,298 119529 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/2
2023-10-18 13:50:33,299 119530 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/0
2023-10-18 13:50:33,299 119530 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/1
2023-10-18 13:50:33,299 119530 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/1/2
2023-10-18 13:50:33,299 119530 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/2/0
...
...finally...
2023-10-18 13:53:45,102 311333 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 13:53:45,102 311333 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 13:53:45,104 311335 [ main] INFO ome.formats.importer.ImportCandidates - 444 file(s) parsed into 0 group(s) with 433 call(s) to setId in 306519ms. (310440ms total) [0 unknowns]
2023-10-18 13:53:45,153 311384 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-18 13:53:45,692 311923 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-18 13:53:46,250 312481 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-18 13:53:46,269 312500 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.99.1.el7.x86_64
No imports found
Import metadata-only NGFF without chunks...
(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels_mo.log --errs /tmp/idr0054_perf_nopixels_mo.err
...
No import found
Finally, testing OMEZarrReader_performance_allfiles.jar
...
(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels.log --errs /tmp/idr0054_perf_nopixels.err
...
2023-10-18 14:28:25,402 325332 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 14:28:25,659 325589 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-18 14:28:32,274 332204 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 14:28:32,464 332394 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61516644
2023-10-18 14:28:34,758 334688 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 61516644
2023-10-18 14:28:41,556 341486 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 61516644
2023-10-18 14:28:42,425 342355 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 61516644
2023-10-18 14:28:42,481 342411 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 61516644
2023-10-18 14:28:42,527 342457 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61516644
2023-10-18 14:28:42,722 342652 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314541
==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:05:35.200
metadata-only...
(base) (venv3) [wmoore@test120-omeroreadwrite ~]$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_perf_nopixels_mo.log --errs /tmp/idr0054_perf_nopixels_mo.err
...
2023-10-18 14:32:54,532 9218 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-10-18 14:32:54,810 9496 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-18 14:32:54,915 9601 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61517100
2023-10-18 14:32:55,420 10106 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 61517100
2023-10-18 14:33:00,874 15560 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 61517100
2023-10-18 14:33:01,498 16184 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 61517100
2023-10-18 14:33:01,544 16230 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 61517100
2023-10-18 14:33:01,585 16271 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61517100
2023-10-18 14:33:01,785 16471 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /home/wmoore/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314542
==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:14.078
Black image but import OK.
As discussed with @dgault... The allfiles
ZarrReaders above fail to create memo file for mkngff
Filesets, although they can import NGFF OK. Question to test: can these allfiles
ZarrReaders create a memo file for NGFF filesets created by regular import (if the memo file is deleted)?
We still are using OMEZarrReader_performance_allfiles.jar
from yesterday...
Using the 4 images successfully imported yesterday with allfiles
ZarrReaders (with/without performace PR, with/without chunks)
(base) [wmoore@test120-omeroreadwrite ~]$ grep -A 2 "13-18-30.948/a0e1b9c0-5c07-" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.1 | grep -A 2 "saved memo"
2023-10-18 13:26:31,122 DEBUG [ loci.formats.Memoizer] (2-thread-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/.0.bfmemo (46364 bytes)
2023-10-18 13:26:31,122 DEBUG [ loci.formats.Memoizer] (2-thread-3) start[1697635590735] time[387] tag[loci.formats.Memoizer.setId]
(base) [wmoore@test120-omeroreadwrite ~]$ sudo rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/.0.bfmemo
The image was still viewable in the webclient and a new `.0.bfmemo` file has been created, although there's nothing new in the Blitz.log when searching with `13-18-30.948/a0e1b9c0-5c07-` as above.
(base) [wmoore@test120-omeroreadwrite ~]$ sudo ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/ total 40K drwxrwxr-x. 2 omero-server omero-server 31 Oct 19 09:05 . drwxrwxr-x. 3 omero-server omero-server 15 Oct 18 13:26 .. -rw-rw-r--. 1 omero-server omero-server 37K Oct 19 09:05 .0.bfmemo
Repeated with exactly the same behaviour for the other 3 images.
I noticed a change in behaviour with this PR...
In the testing on idr-testing of various builds above https://github.com/ome/ZarrReader/pull/64#issuecomment-1768612367 I have imported from s3 bucket like this for a couple of tests...
omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
With OMEZarrReader_canonical_allfiles.jar
I got Fileset: 6314491
. This has ManagedRepo paths like:
demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/2/0/21/0/0/0
demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/18/13-18-30.948/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4/0/10/0/0/0
With the OMEZarrReader_performance_allfiles.jar
I got Fileset: 6314541
. The clientpath
is the same as above, but the ManagedRepo paths are much longer, since they contain the full path: (extra bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39
in each path):
demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/18/14-23-08.177/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1/0/8/0/1/0
demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/18/14-23-08.177/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1/0/8/0/1/1
I noticed this originally at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1778121070
Another issue importing with this PR: Channels - see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1779595416
EDIT: This doesn't happen with OMEZarrReader-0.3.1-jar-with-dependencies.jar
but does with OMEZarrReader_performance_allfiles.jar
.
https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1777374620
Updated the reader to allow setting of the list pixels flag via an environment variable (OME_ZARR_LIST_PIXELS).
Also added an additional method for reloading the options file. Though at the minute it seems that the cache directory which we had hoped to use is only available at the Memoizer level. So we will need to find a way to get that cache location into the reader
Updated with a commit that will force the reader to attempt to locate a bfoptions from the same location as the top level folder. Local testing shows this working even when the top level folder is a symlink. This will be the location it checks no matter which file within the dataset is initialized. If your top level folder is called myData.ome.zarr
then you will need an options file beside it called myData.ome.zarr.bfoptions
.
The options file itself should simply contain:
omezarr.list_pixels=false
Testing today's build on idr-testing
...
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/586/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p664_b586.jar
...etc, retstart...
Look for idr0091
images that don't already have memo file created - generates ResourceError:
http://localhost:1080/webclient/?show=image-10648049
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zattrs
As omero-server
...
echo "omezarr.list_pixels=false" > /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions
check...
$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff
total 8.0K
drwxr-sr-x. 2 omero-server omero-server 114 Oct 31 10:15 .
drwxrwsr-x. 25 omero-server omero-server 4.0K Oct 11 09:49 ..
lrwxrwxrwx. 1 omero-server omero-server 109 Oct 11 09:46 d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr -> /bia-integrator-data/S-BIAD852/d28d77b1-5807-47fc-b3a6-57e58b93632f/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr
-rw-r--r--. 1 omero-server omero-server 26 Oct 31 10:15 d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions
When trying to view the image again, unfortunately I still see an error... Blitz logs...
2023-10-31 10:20:29,917 WARN [ loci.formats.FormatHandler] (l.Server-9) Mismatch with group key paths and original OME-XML metadata, original ordering wont be maintained
2023-10-31 10:20:33,227 ERROR [ loci.formats.FormatHandler] (l.Server-9) ZarrReader loaded options file from: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr.bfoptions
2023-10-31 10:20:33,227 ERROR [ loci.formats.FormatHandler] (l.Server-9) ZarrReader getUsed files, skipPixels: false
2023-10-31 10:20:33,228 ERROR [ loci.formats.FormatHandler] (l.Server-9) ZarrReader fetching list of used files: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr
2023-10-31 10:20:33,228 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zattrs
2023-10-31 10:20:33,228 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/.zgroup
2023-10-31 10:20:33,229 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/0/.zarray
2023-10-31 10:20:33,231 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/0/0/0/0/0
...
...
58b93632f.zarr/3/98/2/0/0
2023-10-31 10:20:37,032 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/0/0/0
2023-10-31 10:20:37,032 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/1/0/0
2023-10-31 10:20:37,033 ERROR [ loci.formats.FormatHandler] (l.Server-9) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e
58b93632f.zarr/3/99/2/0/0
2023-10-31 10:20:37,058 WARN [ o.s.b.repo.ManagedReaderSecurityCheck] (l.Server-9) file demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/3/417/0/0/0 does not have a unique readab
le fileset
2023-10-31 10:20:37,058 DEBUG [ loci.formats.Memoizer] (l.Server-9) start[1698747619235] time[17823] tag[loci.formats.Memoizer.setId]
2023-10-31 10:20:37,059 ERROR [ ome.io.bioformats.BfPixelBuffer] (l.Server-9) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-4
7fc-b3a6-57e58b93632f.zarr/.zattrs
2023-10-31 10:20:37,060 ERROR [ ome.io.nio.PixelsService] (l.Server-9) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-
57e58b93632f.zarr/.zattrs
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-19/2020-10/02/14-57-03.867_mkngff/d28d77b1-5807-47fc-b3a6-57e58b93632f.zarr/3/417/0/0/0 does not have a unique readable fileset
at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
at jdk.internal.reflect.GeneratedMethodAccessor316.invoke(Unknown Source)
With that last fix, updated ZarrReader and the Image is viewable! 👍
Woo hoo! 🎉
Testing import on idr-testing with the bfoptions supported ZarrReader...
Import idr0054 sample image as above - regular import (including chunks) from s3...
omero import -d 18153 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_bfoptions.log --errs /tmp/idr0054_bfoptions.err
...
2023-11-01 15:49:26,804 343015 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0/0/0/0/0/0
Other imported objects:
Fileset:6314593
==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:05:29.496
Updated the proxy idr-testing omeroreadonly-X
servers with same OMEZarrReader_p665_b587.jar
as tested above on idr-testing omeroreadwrite
server.
Testing whether that last commit fixes issue with mismatching pixel values on multi-well plates (that contain OME/METADATA.ome.xml) as reported https://github.com/IDR/idr-metadata/issues/675#issuecomment-1794822262
Updated ZarrReader on idr-testing to today's build jar named OMEZarrReader_p672_b594.jar
The Channels issue above seems to be fixed 👍
However, ManagedRepo paths issue remains (see https://github.com/ome/ZarrReader/pull/64#issuecomment-1780650291)
Importing idr0125 Plates e.g. original data is at /ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr
on idr0125-pilot, I get paths like this (which shouldn't contain ngff/idr0125/
):
demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-11/07/09-32-28.422/ngff/idr0125/SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr/C/10/8/.zgroup
@will-moore, with that last commit I believe the file path issue should be resolved. It seems that getRequiredDirectories
is used to trim the paths down to remove the common path, see https://github.com/ome/omero-blitz/blob/8d18dfcecc63b002ad89e924e6f03763b16052de/src/main/java/ome/services/blitz/repo/ManagedRepositoryI.java#L1467. For Zarr files we should only need one level below the common parent, that being the .zarr
folder itself.
Initial performance improvements for ZarrReader related to https://github.com/ome/ZarrReader/issues/63
This PR is an initial PR for testing purposes, in its current state it looks to reduce the initFile time by about 50%, hopefully further improvements can be made.
Based on a subset of idr0036 the initial profiling looked as below,
The updated profiling should look more like this (the Zarrutils.toJSON should also be gone). If we can manage to remove some of the remaining JZarr calls then this may be reduced further:
The initial fixes here target the below areas: