Closed dgault closed 3 months ago
jburel
commented
6 months ago The dependency was removed in the first instance since it was breaking the openmicroscopy build.
Re-introducing it will solve one problem but not the other. This will require some investigation, the problem noticed at the time might be fixed in newer version of the aws-sdk-java library.
The version added by this PR is 4 years old. We should should try to add a more recent version of the library (latest is 2.25.7)
dgault
commented
6 months ago Some initial performance testing results. First testing idr0090 plate 190206, comparing the local filesystem to s3, both via the standard setId and openBytes combination, and then via the new static openPlane function. Each test result is averaged from 10 runs
From a local filesystem via setID (with memo):
| SetID (with memo) | openBytes | Total time | |
|---|---|---|---|
| Average time (ms): | 1781.838209 | 71.2004399 | 1853.038649 |
From a local filesystem via the new static method
| new openPlane | |
|---|---|
| Average time (ms): | 634.9016599 |
From s3 via setID (with memo) using the new bfoptions method:
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr| SetID (with memo) | openBytes | Total time | |
|---|---|---|---|
| Average time (ms): | 1291.234305 | 3295.094909 | 4586.329215 |
Surprisingly in this case the setID time was consistenly lower which I would not have expected
From s3 via the new static method
ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr/B/2/0/0", newBuffer);| new openPlane | |
|---|---|
| Average time (ms): | 5133.552719 |
Profiling the calls
I had expected the calls to openBytes and openPlane to be closer to that of the filesystem so I profiled the new static openPlane for both the local filesystem and s3 calls.
Local filesystem:
s3:
Interestingly the Blosc decompression appears to be considerably slower when using the s3 option
idr0054
I also tested idr0054 (same as Seb had used for his profiling), 10 runs
Using the new static openPlane function from s3
ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0", newBuffer);Average time (ms): 4213.232465
Profiling the call shows about 50% of the time being taken by the BloscCompressor uncompressing the data:
dgault
commented
6 months ago With the build on https://github.com/ome/openmicroscopy/pull/6379 now green hopefully this PR should be ready for some integration testing.
As far as next steps, there are 2 options for testing, using the bfoptions to load direct from S3 or using the new static method and bypassing the reader initialisation and loading of the memo file.
To load directly from S3 the path should be added to the bfoptions file using the new option. This will also require the memo files to be re-generated. Note that in testing the memo files were not detected as being invalid so I had to delete them first. An example of how this would look in the bfoptions file:
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarrTo test the new static method will require changes to how pixel data is retrieved. Currently this is specific to the ZarrReader and any non Zarr data will not be able to use this approach. Instead of initialising the reader via the Memoizer and calling openBytes you can simply call the new method, pointing to the location of a zarray that you wish to open. A buffer needs to be declared first also.
ZarrReader.openPlane("https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0", newBuffer);Removed the exclude flag
snoopycrimecop
commented
5 months ago Conflicting PR. Removed from build BIOFORMATS-push#4. See the console output for more details. Possible conflicts:
--conflicts
snoopycrimecop
commented
5 months ago Conflicting PR. Removed from build BIOFORMATS-push#808. See the console output for more details. Possible conflicts:
--conflicts
snoopycrimecop
commented
5 months ago Conflicting PR. Removed from build BIOFORMATS-push#5. See the console output for more details. Possible conflicts:
--conflicts
snoopycrimecop
commented
5 months ago Conflicting PR. Removed from build BIOFORMATS-push#809. See the console output for more details. Possible conflicts:
--conflicts Conflict resolved in build BIOFORMATS-push#6. See the console output for more details.
will-moore
commented
5 months ago Testing today's build... Merged in https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/6/console
$ ssh -A idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80
$ ssh omeroreadwrite
$ sudo -u omero-server -s
$ cd
$ wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/label=testintegration/7/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
$ mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p6_b7.jar
$ rm OMERO.server/lib/client/OMEZarrReader.jar
$ rm OMERO.server/lib/server/OMEZarrReader.jar
$ cp OMEZarrReader_p6_b7.jar OMERO.server/lib/client/
$ cp OMEZarrReader_p6_b7.jar OMERO.server/lib/server/restart server...
Tested render_image performance on idr0010 - 1st plate using existing memo file (same behaviour as without this PR)
Then deleted memo file and added bfoption...
rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo$ cat /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e7bdccbf-5119-4d9c-b466-66d211871db8.zarrView image to regenerate memo... 13:45 -> approx 10 minutes...
Timed the render_image_region (whole plane) on idr-testing:omeroreadwrite (no microservices) via localhost:1080, selecting a Well each time, with the Preview panel open.
Average approx 2 seconds with the goofys access - green (if we ignore the outliers > 3.5 secs) and average about 1 second with this PR - blue:
will-moore
commented
5 months ago Repeat the testing above on idr0090 - first plate:
bash-4.2$ cat /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2021-02/18/20-50-17.861_mkngff/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/a5a2714b-bfbf-4251-95ac-5319fda4bf69/a5a2714b-bfbf-4251-95ac-5319fda4bf69.zarrmemo started 15:06...
will-moore
commented
5 months ago It seems that the idr0090 plate above is still not viewable. Grepping the logs for Fileset path finds a lot of repetition...
will-moore
commented
5 months ago Going to try on a different study - idr0016, testing the memo file regeneration with and without the alt_store setting in bfoptions....
Delete the memo files for first 2 plates from idr0016...
bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-08/18/18-11-19.764_mkngff/171961d7-6cca-4201-87e4-449ff3880d48.zarr/..zattrs.bfmemo
bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-06/15/22-36-23.585_mkngff/f1a84265-b1fa-4ef5-b8fa-fb0c01e2c24e.zarr/..zattrs.bfmemo Update the bfoptions of the first plate only...
$ cat /data/OMERO/ManagedRepository/demo_2/2017-08/18/18-11-19.764_mkngff/171961d7-6cca-4201-87e4-449ff3880d48.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/171961d7-6cca-4201-87e4-449ff3880d48/171961d7-6cca-4201-87e4-449ff3880d48.zarrThen view images to trigger memo file...
After ~40 mins I noticed lots of DatabaseBusyException in the Blitz logs. Seems the server is overwhelmed trying to create memos for 2 plates from idr0016 (and still trying for the idr0090 plate above)?
Restarted the server... ~ 10:50
Compared time to render_image when initially loading each Well in Preview panel:
will-moore
commented
5 months ago Comparing the log files from idr0090 with the regeneration of idr0010 memo file above, idr0010 looks like this at the time that "memo saved" (there was no "memo saved" seen yet for idr0090 logs above):
idr0010: $ grep "e7bdccbf-5119-4d9c-b466-66d211871db8" OMERO.server/var/log/Blitz-0.log...
...
2024-03-19 13:36:10,073 INFO [ ome.io.nio.PixelsService] (l.Server-0) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 66
2024-03-19 13:36:10,481 INFO [ ome.services.OmeroFilePathResolver] (l.Server-9) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:36:10,513 DEBUG [ loci.formats.Memoizer] (l.Server-9) loaded memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (520047 bytes)
2024-03-19 13:36:10,515 INFO [ ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml Series: 66
2024-03-19 13:36:58,664 INFO [ ome.services.util.ServiceHandler] (l.Server-4) Rslt: ([demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/, .zattrs, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], [demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], [demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/A/, .zgroup, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD885/e7bdccbf-5119-4d9c-b466-66d211871db8/e], ... 2702 more)
2024-03-19 13:44:48,528 INFO [ ome.services.OmeroFilePathResolver] (l.Server-1) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:44:49,615 INFO [ loci.formats.ImageReader] (l.Server-1) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:52:15,347 INFO [ ome.services.OmeroFilePathResolver] (l.Server-5) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:52:15,561 INFO [ loci.formats.ImageReader] (l.Server-5) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/METADATA.ome.xml
2024-03-19 13:57:24,720 DEBUG [ loci.formats.Memoizer] (l.Server-1) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo17125964820390949463
2024-03-19 13:57:24,720 DEBUG [ loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/21/00-22-55.102_mkngff/e7bdccbf-5119-4d9c-b466-66d211871db8.zarr/OME/.METADATA.ome.xml.bfmemo (520210 bytes)The line with ome.services.util.ServiceHandler] (l.Server-4) Rslt: .... ... 2702 more) is followed closely by memo saved but for idr0090 logs above, the Rslt.... ... 12646 more) is not followed by memo saved.
will-moore
commented
5 months ago Re-started memo file generation for idr0090 (first plate) by clicking on Preview panel....
Worked this time: On idr-testing:readonly server (no microservices as above). render_image_region times for initial Well loaded in Preview panel for idr0090 first plate (s3 vv goofys):
will-moore
commented
5 months ago As expected, once we have memo file generated with the alt_store option, we can unmount the goofys file system and continue to render images.
However, less expected is that if we unmount goofys before the memo file is re-generated then we get a ResourceError.
Using these commands on idr-testing...
sudo umount /bia-integrator-data
sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-dataUpdating ZarrReader on all omeroreadonly servers on idr-testing... Ran this on each as omero-server...
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/label=testintegration/16/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_b16.jar
$ rm OMERO.server/lib/client/OMEZarrReader.jar
$ rm OMERO.server/lib/server/OMEZarrReader.jar
$ cp OMEZarrReader_b16.jar OMERO.server/lib/client/
$ cp OMEZarrReader_b16.jar OMERO.server/lib/server/Then restarted...
will-moore
commented
5 months ago I have added the alt_source to the .bfoptions for all NGFF filesets on idr-testing with script and commands at https://github.com/IDR/mkngff_upgrade_scripts/pull/1
will-moore
commented
5 months ago Started memo file generation for all NGFF filesets on idr-testing, using ngff_ids.txt from https://github.com/IDR/idr-metadata/issues/686#issuecomment-1953936897
$ screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240326/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'EDIT: 27th March 15:12 - getting DatabaseBusyExceptions.
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240326/1/**/* | wc
407 1221 32202Cancel screen, Restart all servers":
for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done21:14: restarted...
screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240327/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'9:00 on 28th - current status: 647 Images (Plates) processed, 576 of them are "ok" out of a total of 6763 in the total list (doesn't include Datasets).
[wmoore@test120-proxy ~]$ ls /tmp/ngff_cache_20240327/1/ | wc
647 647 9012
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
576 1728 45506
[wmoore@test120-proxy ~]$ grep Image ids.txt | wc
6763 6763 9587710:30 pm on 29th March:
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
758 2274 5998010:00 pm on 1st April:
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240327/1/**/* | wc
1107 3321 87656It feels like memo file generation is taking longer than before, but I don't have comparable numbers...
Since we have some timings for idr0090 memo generation at https://github.com/IDR/idr-metadata/issues/686#issuecomment-1958959416, and it would be useful to have the memo files for testing idr0090, let's cancel running above and run idr0090 only...
Status before cancelling...
screen -r
...
Computer:jobs running/jobs completed/%of started jobs
ETA: 155479s Left: 431 AVG: 360.76s 1:10/152/12%/2877.0s 2:10/196/16%/2231.1s 3:10/449/36%/973.9s 4:10/251/20%/1742.2s 5:10/164/13%/2666.4s Restarted on idr0090 only...
screen -dmS cache parallel --eta --sshloginfile nodes -a idr0090_ids.txt --results /tmp/ngff_cache_20240401/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'After 7 hours:
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240401/1/**/*
/tmp/ngff_cache_20240401/1/Image:12542037/stdout:ok: 12542037 11.96731948852539
/tmp/ngff_cache_20240401/1/Image:12546037/stdout:ok: 12546037 4475.028658628464
/tmp/ngff_cache_20240401/1/Image:12548245/stdout:ok: 12548245 4676.609424352646
/tmp/ngff_cache_20240401/1/Image:12552053/stdout:ok: 12552053 4447.661913156509
/tmp/ngff_cache_20240401/1/Image:12554709/stdout:ok: 12554709 3401.6648321151733 These times (avg 4249 if we omit the 11.96 where the memo already exists) look comparable to times at https://github.com/IDR/idr-metadata/issues/686#issuecomment-1958959416
which were 5772.45, 2015.81, 5618.20, 3435.90, 3453.46 (avg 4059).
Still running....
will-moore
commented
5 months ago Are those recent V1 AWS SDK changes going to need a new memo file generation, or will the previous memo files be OK with the new ZarrReader jar?
will-moore
commented
5 months ago Re-running memo for all ngff on idr-testing...
[wmoore@test120-proxy ~]$ screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240405/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'Memo regeneration completed, but new ZarrReader jar will need new memo files (won't be detected automatically so we need to delete/move old memos).
On idr-testing: omeroreadwrite...
[wmoore@test120-omeroreadwrite ~]$ cd /data/BioFormatsCache/
[wmoore@test120-omeroreadwrite BioFormatsCache]$ ls
data
[wmoore@test120-omeroreadwrite BioFormatsCache]$ sudo -u omero-server -s
$ mv data data_old2Updated all idr-testing readwrite and readonly servers with today's build from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/label=testintegration/30/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
Then triggered memo file generation for idr0012 to give a set of plates to test...
[wmoore@test120-proxy ~]$ screen -dmS cache parallel --eta --sshloginfile nodes -a idr0012_ids.txt --results /tmp/ngff_cache_20240408/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'EDIT: 22:55 - 8th April
screen -r
...
ETA: 464s Left: 1 AVG: 498.31s 1:0/19/27%/1759.1s 2:1/9/14%/3713.7s 3:0/10/14%/3342.3s 4:0/10/14%/3342.3s 5:0/19/27%/1759.1sbut logs show all done (some failures).
Cancel screen and start idr0010....
screen -dmS cache parallel --eta --sshloginfile nodes -a idr0010_ids.txt --results /tmp/ngff_cache_20240408_idr0010/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'10:00 on 9th: Run idr0011B and idr0090 together:
screen -dmS cache parallel --eta --sshloginfile nodes -a idr0011B_idr0090.txt --results /tmp/ngff_cache_20240409_idr0011B_idr0090/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'(then repeated with logs _2 - still some failures)
Run all ngff... 17:50 9th April...
[wmoore@test120-proxy ~]$ screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240409_ngff/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'10th April - cancelled the memo generation at 9:32 to allow web testing of NGFF images on idr-testing, but it seems that performance found in testing was still impacted, since memo generation continued during testing. See https://github.com/IDR/idr-metadata/issues/692
11th April - restarted with ngff_ids.txt
screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240411_ngff/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'Memo generation seems to have halted earlier this (Monday) morning 15th. No new logs being created. Counting success, only about 2/3 of logs are "ok". Connection issues from when we started..
[wmoore@test120-proxy ~]$ ls -alh /tmp/ngff_cache_20240411_ngff/1/ | wc
1486 13367 92011
[wmoore@test120-proxy ~]$ grep "ok:" /tmp/ngff_cache_20240411_ngff/1/**/* | wc
1073 3219 90437
[wmoore@test120-proxy ~]$ cat /tmp/ngff_cache_20240411_ngff/1/Image\:692223/stderr
WARNING:omero.client:ome.conditions.DatabaseBusyException:ConcurrencyException: cannot create transaction
Please retry in 5000ms. Cause: cannot create transaction - createSession retry: 1
Internal error. Please contact your administrator:
Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection
stdin is not a terminal: cannot request password
InternalException: Failed to connect: die called: stdin is not a terminal: cannot request password
[wmoore@test120-proxy ~]$ grep "rror" /tmp/ngff_cache_20240411_ngff/1/**/* | wc
341 2263 38901Lots of connection errors, DatabaseBusyExceptions, a few Resource Errors (idr0013 and idr0012) but many of these the Images are viewable. No obvious issues that need fixing. Just re-run...
screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_ids.txt --results /tmp/ngff_cache_20240415_ngff/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'16th April. To check whether a study is complete... Make a list of all "ok" images, then grep that list for each Image in study:
grep "ok:" /tmp/ngff_cache_20240415_ngff/1/**/* > ok_20240415_ngff.txt
for i in $(cat idr0090_ids.txt); do echo $i && grep $i ok_20240415_ngff.txt; doneE.g. All Images are done for idr0004, idr0010, idr00012, idr0013B, idr00015, idr00016 (except last ~20 plates), idr0025, idr0036, idr0090 - NOT all of idr0035, or idr0064.
Cancelled screen at 7:00 - 16th April
Run just for Dataset studies, 16th April, 12:13...
screen -dmS cache parallel --eta --sshloginfile nodes -a ngff_dataset_ids.txt --results /tmp/ngff_cache_20240416_ngff/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'17th April: Ran idr0035 and idr0064 overnight last night. Any images that were not "ok" I manually checked/triggered in webclient and are now OK.
19th April - 23:20 - started ALL memo file generation running
dgault
commented
4 months ago PR should be ready for a final review now. The main goal of the PR was to allow IDR to read directly from Embassy S3 storage. This requires the endpoint configuration option to be set which meant that I was unable to use the same approach as recommended by JZarr https://jzarr.readthedocs.io/en/latest/amazonS3.html. Long term this should likely be replaced with a more centralized approach, allowing for a single point for managing the S3 access. Also for IDR's purpose the endpoint is stored in the bfoptions file alongside the dataset, which required the new alt_store key to be added.
Initially the PR explored 2 options, 1 using the standard set|id and openBytes, the other was adding static methods to read directly from a given path. The methods added for the second option (here) are currently not being used and can be removed from this PR to simplify its includion.
will-moore
commented
4 months ago I tried testing whether the goofys mount on idr-testing was still being used by anything, or if this PR completely replaces access via the filesystem...
Testing only on omeroreadwrite:
$ ssh -A idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80
$ sudo umount /bia-integrator-dataThen browsed all NGFF data at http://localhost:1080/webclient/
All studies were browsable without errors except most of the idr0010 plates. Investigation revealed that the .bfoptions was missing the alt_store:
[wmoore@test120-omeroreadwrite ~]$ cat /data/OMERO/ManagedRepository/demo_2/2016-10/14/05-44-15.120_mkngff/8008bd41-f7f5-4e03-9129-fcf8218bc46e.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=trueSo, this confirms that the goofys mount is NOT needed for viewing images with this PR
melissalinkert
commented
4 months ago Before I start reviewing in depth, what's the best way to test this outside of IDR? Is it sufficient to just add the reader jar to the CLASSPATH and try showinf on a few local/s3 zarrs?
dgault
commented
4 months ago Yeah that should be sufficient to test, or using the 'with dependencies' jar from the build might be easiest.
melissalinkert
commented
4 months ago I think I'm maybe misunderstanding how to evaluate these changes then. With OMEZarrReader-0.4.1-jar-with-dependencies.jar added to the CLASSPATH and a local dataset nucleus3.zarr (generated from test&sizeX=10000&sizeY=10000&sizeC=3.fake with bioformats2raw 0.9.2):
$ showinf -no-upgrade nucleus3.zarr
Exception in thread "main" java.io.FileNotFoundException: nucleus3.zarr (Is a directory)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:130)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151)
at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165)
at loci.common.Location.getHandle(Location.java:522)
at loci.common.Location.getHandle(Location.java:462)
at loci.common.Location.getHandle(Location.java:443)
at loci.common.Location.getHandle(Location.java:426)
at loci.common.Location.checkValidId(Location.java:551)
at loci.formats.ImageReader.getReader(ImageReader.java:184)
at loci.formats.ImageReader.getFormat(ImageReader.java:164)
at loci.formats.tools.ImageInfo.configureReaderPreInit(ImageInfo.java:440)
at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1038)
at loci.formats.tools.ImageInfo.main(ImageInfo.java:1129)which suggests that showinf is not a suitable way to test (I tried picking specific files as well, and just got UnknownFormatException instead).
Using some simple test code:
import loci.formats.MetadataTools;
import loci.formats.meta.IMetadata;
import loci.formats.in.ZarrReader;
public class ZarrReaderTest {
public static void main(String[] args) throws Exception {
ZarrReader reader = new ZarrReader();
try {
IMetadata metadata = MetadataTools.createOMEXMLMetadata();
reader.setMetadataStore(metadata);
reader.setFlattenedResolutions(false);
reader.setId(args[0]);
System.out.println("# series = " + reader.getSeriesCount());
for (int s=0; s<reader.getSeriesCount(); s++) {
reader.setSeries(s);
System.out.println("series #" + s + ":");
System.out.println(" # resolutions = " + reader.getResolutionCount());
for (int r=0; r<reader.getResolutionCount(); r++) {
reader.setResolution(r);
System.out.println(" resolution #" + r + ":");
System.out.println(" X = " + reader.getSizeX());
System.out.println(" Y = " + reader.getSizeY());
System.out.println(" image count = " + reader.getImageCount());
for (int i=0; i<reader.getImageCount(); i++) {
long t0 = System.currentTimeMillis();
reader.openBytes(i, 0, 0, 1, 1);
long t1 = System.currentTimeMillis();
System.out.println(" reading one pixel from plane #" + i + " took " + (t1-t0) + " ms");
}
}
}
}
finally {
reader.close();
}
}
}then running java -mx2g ZarrReaderTest nucleus3.zarr on the same dataset prints the dimensions I'd expect.
Running the same test code on a s3 dataset referenced above:
$ java -mx2g ZarrReaderTest s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:35.447 [main] DEBUG loci.common.services.ServiceFactory -- Loaded properties from: services.properties
13:25:35.463 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface ome.codecs.services.JAIIIOService and implementation class ome.codecs.services.JAIIIOServiceImpl
13:25:35.467 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGXRService and implementation class loci.formats.services.JPEGXRServiceImpl
13:25:35.468 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MDBService and implementation class loci.formats.services.MDBServiceImpl
13:25:35.469 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MetakitService and implementation class loci.formats.services.MetakitServiceImpl
13:25:35.469 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.EXIFService and implementation class loci.formats.services.EXIFServiceImpl
13:25:35.470 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JHDFService and implementation class loci.formats.services.JHDFServiceImpl
13:25:35.471 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.NetCDFService and implementation class loci.formats.services.NetCDFServiceImpl
13:25:35.472 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JAIIIOService and implementation class loci.formats.services.JAIIIOServiceImpl
13:25:35.481 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.OMEXMLService and implementation class loci.formats.services.OMEXMLServiceImpl
13:25:35.483 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.POIService and implementation class loci.formats.services.POIServiceImpl
13:25:35.483 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.common.services.S3ClientService and implementation class loci.common.services.S3ClientServiceImpl
13:25:35.495 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGTurboService and implementation class loci.formats.services.JPEGTurboServiceImpl
13:25:35.772 [main] DEBUG loci.formats.FormatHandler -- ZarrReader initializing s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:35.772 [main] DEBUG loci.formats.FormatHandler -- loci.formats.in.ZarrReader.initFile(s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr)
13:25:52.754 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx000000000000000f46227-006633daae-23b77beb-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182536Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx000000000000000f46227-006633daae-23b77beb-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:25:50 GMT
Connection: Keep-Alive
}
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
at loci.common.S3Handle.initialize(S3Handle.java:275)
at loci.common.S3Handle.<init>(S3Handle.java:228)
at loci.common.S3Handle.<init>(S3Handle.java:138)
at loci.common.Location.getHandle(Location.java:501)
at loci.common.Location.getHandle(Location.java:462)
at loci.common.Location.getHandle(Location.java:443)
at loci.common.Location.getHandle(Location.java:426)
at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
at loci.common.Location.exists(Location.java:768)
at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
at loci.formats.FormatReader.initFile(FormatReader.java:256)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:184)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:13)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
at io.minio.MinioClient.execute(MinioClient.java:1134)
at io.minio.MinioClient.executeHead(MinioClient.java:1240)
at io.minio.MinioClient.statObject(MinioClient.java:1384)
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
... 14 common frames omitted
13:25:52.763 [main] DEBUG loci.formats.FormatHandler -- ZarrReader attempting to initialize file: s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:25:53.501 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx0000000000000010599b6-006633daaf-23b74c3c-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182552Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx0000000000000010599b6-006633daaf-23b74c3c-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:25:51 GMT
Connection: Keep-Alive
}
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
at loci.common.S3Handle.initialize(S3Handle.java:275)
at loci.common.S3Handle.<init>(S3Handle.java:228)
at loci.common.S3Handle.<init>(S3Handle.java:138)
at loci.common.Location.getHandle(Location.java:501)
at loci.common.Location.getHandle(Location.java:462)
at loci.common.Location.getHandle(Location.java:443)
at loci.common.Location.getHandle(Location.java:426)
at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
at loci.common.Location.exists(Location.java:768)
at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
at loci.formats.in.ZarrReader.reloadOptionsFile(ZarrReader.java:1238)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:195)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:13)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
at io.minio.MinioClient.execute(MinioClient.java:1134)
at io.minio.MinioClient.executeHead(MinioClient.java:1240)
at io.minio.MinioClient.statObject(MinioClient.java:1384)
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
... 14 common frames omitted
Exception in thread "main" java.io.IOException: Path 's3:/uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr' is not a valid path or not a directory.
at com.bc.zarr.ZarrUtils.ensureDirectory(ZarrUtils.java:158)
at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:88)
at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:114)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:203)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:13)If I then update the code to set the omezarr.alt_store reader option as in https://github.com/ome/ZarrReader/pull/82#issuecomment-1997609460:
import loci.formats.MetadataTools;
import loci.formats.meta.IMetadata;
import loci.formats.in.DynamicMetadataOptions;
import loci.formats.in.ZarrReader;
public class ZarrReaderTest {
public static void main(String[] args) throws Exception {
ZarrReader reader = new ZarrReader();
DynamicMetadataOptions options = (DynamicMetadataOptions) reader.getMetadataOptions();
options.set("omezarr.alt_store", "https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr");
reader.setMetadataOptions(options);
try {
IMetadata metadata = MetadataTools.createOMEXMLMetadata();
reader.setMetadataStore(metadata);
reader.setFlattenedResolutions(false);
reader.setId(args[0]);
System.out.println("# series = " + reader.getSeriesCount());
for (int s=0; s<reader.getSeriesCount(); s++) {
reader.setSeries(s);
System.out.println("series #" + s + ":");
System.out.println(" # resolutions = " + reader.getResolutionCount());
for (int r=0; r<reader.getResolutionCount(); r++) {
reader.setResolution(r);
System.out.println(" resolution #" + r + ":");
System.out.println(" X = " + reader.getSizeX());
System.out.println(" Y = " + reader.getSizeY());
System.out.println(" image count = " + reader.getImageCount());
for (int i=0; i<reader.getImageCount(); i++) {
long t0 = System.currentTimeMillis();
reader.openBytes(i, 0, 0, 1, 1);
long t1 = System.currentTimeMillis();
System.out.println(" reading one pixel from plane #" + i + " took " + (t1-t0) + " ms");
}
}
}
}
finally {
reader.close();
}
}
}and retry:
$ java -mx2g ZarrReaderTest s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:17.511 [main] DEBUG loci.common.services.ServiceFactory -- Loaded properties from: services.properties
13:28:17.539 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface ome.codecs.services.JAIIIOService and implementation class ome.codecs.services.JAIIIOServiceImpl
13:28:17.539 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGXRService and implementation class loci.formats.services.JPEGXRServiceImpl
13:28:17.540 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MDBService and implementation class loci.formats.services.MDBServiceImpl
13:28:17.541 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.MetakitService and implementation class loci.formats.services.MetakitServiceImpl
13:28:17.541 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.EXIFService and implementation class loci.formats.services.EXIFServiceImpl
13:28:17.542 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JHDFService and implementation class loci.formats.services.JHDFServiceImpl
13:28:17.543 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.NetCDFService and implementation class loci.formats.services.NetCDFServiceImpl
13:28:17.544 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JAIIIOService and implementation class loci.formats.services.JAIIIOServiceImpl
13:28:17.546 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.OMEXMLService and implementation class loci.formats.services.OMEXMLServiceImpl
13:28:17.548 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.POIService and implementation class loci.formats.services.POIServiceImpl
13:28:17.548 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.common.services.S3ClientService and implementation class loci.common.services.S3ClientServiceImpl
13:28:17.556 [main] DEBUG loci.common.services.ServiceFactory -- Added interface interface loci.formats.services.JPEGTurboService and implementation class loci.formats.services.JPEGTurboServiceImpl
13:28:17.811 [main] DEBUG loci.formats.FormatHandler -- ZarrReader initializing s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:17.811 [main] DEBUG loci.formats.FormatHandler -- loci.formats.in.ZarrReader.initFile(s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr)
13:28:35.136 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx000000000000000f46246-006633db50-23b77beb-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182818Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx000000000000000f46246-006633db50-23b77beb-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:28:32 GMT
Connection: Keep-Alive
}
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
at loci.common.S3Handle.initialize(S3Handle.java:275)
at loci.common.S3Handle.<init>(S3Handle.java:228)
at loci.common.S3Handle.<init>(S3Handle.java:138)
at loci.common.Location.getHandle(Location.java:501)
at loci.common.Location.getHandle(Location.java:462)
at loci.common.Location.getHandle(Location.java:443)
at loci.common.Location.getHandle(Location.java:426)
at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
at loci.common.Location.exists(Location.java:768)
at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
at loci.formats.FormatReader.initFile(FormatReader.java:256)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:184)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:17)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
at io.minio.MinioClient.execute(MinioClient.java:1134)
at io.minio.MinioClient.executeHead(MinioClient.java:1240)
at io.minio.MinioClient.statObject(MinioClient.java:1384)
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
... 14 common frames omitted
13:28:35.144 [main] DEBUG loci.formats.FormatHandler -- ZarrReader attempting to initialize file: s3://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
13:28:35.209 [main] WARN loci.formats.services.JZarrServiceImpl -- Zarr Reader is not using S3FileSystemStore as this is currently for use with S3 configured with a https endpoint
13:28:35.805 [main] DEBUG loci.common.S3Handle -- Object not found: [server:https://uk1s3.embassy.ebi.ac.uk port:0 bucket:bia-integrator-data path:S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions found:false] {}
loci.common.services.S3ClientServiceException: error occurred
ErrorResponse(code=NoSuchKey, message=Object does not exist, bucketName=bia-integrator-data, objectName=S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, resource=/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, requestId=tx0000000000000010599cf-006633db51-23b74c3c-default, hostId=null)
request={method=HEAD, url=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr.bfoptions, headers=Host: uk1s3.embassy.ebi.ac.uk
User-Agent: Minio (amd64; amd64) minio-java/dev Bio-Formats/dev
x-amz-date: 20240502T182835Z
}
response={code=404, headers=Content-Length: 234
x-amz-request-id: tx0000000000000010599cf-006633db51-23b74c3c-default
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 02 May 2024 18:28:33 GMT
Connection: Keep-Alive
}
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:100)
at loci.common.S3Handle.initialize(S3Handle.java:275)
at loci.common.S3Handle.<init>(S3Handle.java:228)
at loci.common.S3Handle.<init>(S3Handle.java:138)
at loci.common.Location.getHandle(Location.java:501)
at loci.common.Location.getHandle(Location.java:462)
at loci.common.Location.getHandle(Location.java:443)
at loci.common.Location.getHandle(Location.java:426)
at loci.common.Location$URLLocationProperties.<init>(Location.java:133)
at loci.common.Location.exists(Location.java:768)
at loci.formats.in.DynamicMetadataOptions.loadOptions(DynamicMetadataOptions.java:493)
at loci.formats.in.ZarrReader.reloadOptionsFile(ZarrReader.java:1238)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:195)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:17)
Caused by: io.minio.errors.ErrorResponseException: Object does not exist
at io.minio.MinioClient.execute(MinioClient.java:1134)
at io.minio.MinioClient.executeHead(MinioClient.java:1240)
at io.minio.MinioClient.statObject(MinioClient.java:1384)
at loci.common.services.S3ClientServiceImpl.statObject(S3ClientServiceImpl.java:92)
... 14 common frames omitted
Exception in thread "main" java.io.IOException: Path 's3:/uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr' is not a valid path or not a directory.
at com.bc.zarr.ZarrUtils.ensureDirectory(ZarrUtils.java:158)
at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:88)
at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:114)
at loci.formats.in.ZarrReader.initFile(ZarrReader.java:203)
at loci.formats.FormatReader.setId(FormatReader.java:1480)
at ZarrReaderTest.main(ZarrReaderTest.java:17)That looks like a bfoptions file must exist in s3 no matter what? Or am I misunderstanding how to initialize the reader?
dgault
commented
4 months ago The showinf UnknownFormatException seems like the classpath has likely not been picked up correctly. I normally test bftools by modifying bf.sh to detect the jar but I would have expected the classpath to work.
For this PR (and general usage), I would expect the ID used for setID to be a location on a local filesyetm. If setting through ImageReader rather than ZarrReader then it must be one of the files in the .zarr folder rather than the directory itself.
The real goal of the PR was to find a way to provide access to open s3 data from embassy without having to reimport. It is definitely a specific solution for this unique use case rather than a general purpose solution. So for now the s3 access is very specific for use with the new alt store option with a public https endpoint such as:
https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD882/07f2244a-0fae-4a06-b0e8-6bfa586793d0/07f2244a-0fae-4a06-b0e8-6bfa586793d0.zarr
If we want to add support for more general s3 handling then one option would be using then using an existing s3 filesystem with jzarr such as https://jzarr.readthedocs.io/en/latest/amazonS3.html. Though I would suggest that would be a follow up.
sbesson
commented
4 months ago Tried to construct a minimal test scenario by downloading a public OME-Zarr IDR dataset 6001240.zarr locally i.e.
aws s3 --endpoint-url=https://uk1s3.embassy.ebi.ac.uk cp --recursive s3://idr/zarr/v0.4/idr0062A/6001240.zarr/ 6001240.zarr/ --no-sign-requestWith this PR included, running the following command
% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240.zarr/.zattrs -autoscaleworks as expected, reading the metadata and all planes from the local OME-Zarr dataset
Constructed a secondary OME-Zarr where all the chunks (named 0 and 1) were deleted manually so that the dataset only includes .zarray and .zgroup files. The following
% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscalealso works as I expect, reading the metadata but filling the missing chunks with zeros when navigating through the planes.
Trying to combine this metadata-only OME-Zarr with the alternative store option as introduced in this PR via the command-line -option
% BF_CP=./target/OMEZarrReader-0.4.1-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale -option omezarr.alt_store https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarrstill gives me black planes and the initialization includes the following warning
Zarr Reader is not using S3FileSystemStore as this is currently for use with S3 configured with a https endpointIs that the expectation?
dgault
commented
4 months ago Sorry @sbesson, the test you did is spot on but the result was not as expected. That was a mistake on my part in the last commit, it should be corrected now. Rerunning your last test should display the data as before except being read from embassy.
sbesson
commented
4 months ago With the last commit
% BF_CP=./target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscalestill returns black planes as per the missing chunks but
% BF_CP=./target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar ~/Downloads/bftools/showinf ~/Downloads/6001240_nochunks.zarr/.zattrs -autoscale -option omezarr.alt_store https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarrnow loads the planes from the Embassy object store as expected
Re-adding the S3FileSystemStore from https://github.com/ome/ZarrReader/pull/38 in order to enable reading directly from Embassy S3. This will also add the
awssdkdependency back into the project.Some modifications from the original have made and a new option has been added to allow for an alternative s3 location to set in the
bfoptionsfile.