Testing mkngff on idr-testing

will-moore commented 1 year ago

Seb: "test120 now deployed with OMERO.server and available at https://idr-testing.openmicroscopy.org/". As discussed at IDR meeting 11th September, good to test mkngff with ALL the NGFF filesets:

[x] idr0004 mkngff 2.5 hrs (46 x 100-well-Plate - 3mins per Plate) - setId 1 min per Fileset
[x] idr0010 - 148 x 384-well-Plates - mkngff 9 mins per plate 142/147 done
[x] idr0011 mkngff 3.5 hrs (182 x 48-well-Plate) 1 min per plate) - setId 1min per Plate. Improved to 40-50 secs
[x] idr0012 - 68 x 320-well-Plates (2 fields) - mkngff 6 mins per plate. Memo file took 42 minutes
[x] idr0013 - 538 x 384-well-Plates - mkngff 3 mins per plate x 538 => 26 hours. memo file took 11 minutes x 538 = 4 days
[ ] idr0015 - 84 x 396-well-Plates
[x] idr0016 - 413 x 384-well-Plates (6 fields) mkngff 7.5 mins per Plate - x 413 => 51 hours
[x] idr0025 mkngff 3 x 96-well-Plates. setId 95 mins for a plate, 4 mins for Plate 2, 12 mins for Plate 3.
[x] idr0026 mkngff took 90 mins (111 images). setId took 26 secs for a single image
[x] idr0033 - mkngff took 18 hours for 12 x 384-well-plates (9 fields). memo file took 24 hours for a single Plate, then 12 hours for Plate 2 - Now down to 81 minutes
[x] idr0035 - mkngff took 6.5 hours (55 plates). setId took 1 hour for a single Plate
[x] idr0036 - 20 Plates
[x] idr0051 - 5 Images
[x] idr0054 - 3 Images
[x] idr0090 - 22 Plates
[x] idr0091 - 342 Images - on idr0125-pilot mkngff took few secs each. 28 mins total

View on https://idr-testing.openmicroscopy.org - Updated ZarrReader on 2023.09.25

will-moore commented 1 year ago

ssh idr-testing.openmicroscopy.org -At ssh omeroreadwrite -A

# also run this in another terminal:
ssh idr-testing.openmicroscopy.org -A -L 1080:omeroreadwrite:80

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

Install mkngff latest PR https://github.com/IDR/omero-mkngff/pull/8 into /opt/omero/server/venv3..

sudo -u root -s
source /opt/omero/server/venv3/bin/activate

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'
... Resolved https://github.com/will-moore/omero-mkngff to commit 231431164e3692298864fcc52fb3f0c663c8f595
Successfully installed omero-mkngff-0.1.0.dev0

will-moore commented 1 year ago

Going to test idr0051 (3 images), following https://github.com/IDR/omero-mkngff/issues/2 workflow...

idr0051.csv

idr0054/Tonsil_1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39,1591301
idr0054/Tonsil_2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573,1591302
idr0054/Tonsil_3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3,5025553

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"

                 uuid
--------------------------------------
 01b61552-49fc-4543-bf40-...
(1 row)

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 3;"
                 uuid                 
--------------------------------------
 9dee647a-d1ed-483b-8e67-...
 7b025064-588b-4d17-8eb4-...
 01b61552-49fc-4543-bf40-...
(3 rows)

for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15 // 15-27-40.023 for fileset 1591301
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr -> /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15 // 15-28-44.081 for fileset 1591302
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr -> /bia-integrator-data/S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07 // 23-05-45.568 for fileset 5025553
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr -> /bia-integrator-data/S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr

UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311582
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311583
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311584
(1 row)
COMMIT

will-moore commented 1 year ago

Image above is not viewable... https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/ returns

"Exception": "Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml"

sbesson commented 1 year ago

Was the bucket mounted both on the read-write VM and the read-only VMs (which are the ones accessed by the idr-testing endpoint)?

will-moore commented 1 year ago

Update ZarrReader to include all current PRs, as described... https://github.com/IDR/omero-mkngff/issues/2#issuecomment-1669728243

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
...

will-moore commented 1 year ago

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

sbesson commented 1 year ago

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

If you are departing from the release 0.3.1 artifact was deployed as part of https://github.com/IDR/deployment/pull/380, the same JAR will definitely need to be copied under lib/server on all VMs and the omero-server service restarted accordingly

will-moore commented 1 year ago

Mount bia-integrator-data on all the readonly servers...

ssh -A idr-testing.openmicroscopy.org

for server in omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data"; done

@sbesson What about the creation of symlinks by mkngff? Does that need to happen on multiple servers?

sbesson commented 1 year ago

That should work out of the box as the OMERO binary repository should be shared between the read-write VM and all read-only VMs.

will-moore commented 1 year ago

I updated ZarrReader on omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4 as at https://github.com/IDR/omero-mkngff/issues/2#issuecomment-1669728243:

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

But I'm still seeingError instantiating pixel buffer at https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/

I'was trying to check the Blitz.log on omeroreadwrite at the same time as refreshing that URL, but I don't see anything because I guess this is being handled by a different server. I guess I have to try them in turn...

will-moore commented 1 year ago

On omeroreadonly-2 I saw this $ less /opt/omero/server/OMERO.server/var/log/Blitz-0.log...

2023-09-07 09:44:29,669 INFO  [         ome.security.basic.EventHandler] (l.Server-4)  Auth:    user=52,group=3,event=null(User),sess=bb36fb91-d187-4988-8d0e-ef595314e803
2023-09-07 09:44:29,670 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.
ome.xml
2023-09-07 09:44:30,259 INFO  [                loci.formats.ImageReader] (l.Server-4) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-09-07 09:44:33,844 WARN  [              loci.formats.FormatHandler] (l.Server-4) Mismatch with group key paths and original OME-XML metadata, original ordering wont be maintained
2023-09-07 09:44:34,734 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1694079869675] time[5059] tag[loci.formats.Memoizer.setId]
2023-09-07 09:44:34,735 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-4) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/ME
TADATA.ome.xml
2023-09-07 09:44:34,735 ERROR [                ome.io.nio.PixelsService] (l.Server-4) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.om
e.xml
java.lang.RuntimeException: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.eso
tericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
    at IceInternal.ThreadPool.run(ThreadPool.java:395)
    at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
    at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
    at loci.formats.Memoizer$KryoDeser.<init>(Memoizer.java:125)
    at loci.formats.Memoizer.getDeser(Memoizer.java:789)
    at loci.formats.Memoizer.saveMemo(Memoizer.java:990)
    at loci.formats.Memoizer.setId(Memoizer.java:733)
    at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
    ... 82 common frames omitted
2023-09-07 09:43:18,832 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1694079795148] time[3684] tag[omero.call.exception]

will-moore commented 1 year ago

David (slack) "Kryo will be related to the memoizer, I haven’t seen that particular issue though, perhaps a version mismatch with the dependency? if it is also using the latest merge of Bio-Formats then that may be using different versions"

sbesson commented 1 year ago

One thing worth investigating is that the latest OMERO.server release, 5.6.8, was deployed on test120 as opposed to OMERO 5.6.7 on pilot-idr0125. In terms of underlying library changes, Bio-Formats was upgraded from 6.12.0 to 6.14.0 - see https://www.openmicroscopy.org/2023/07/25/omero-5-6-8.html which in turns included a large update of low-level dependencies and notably kryo - see https://www.openmicroscopy.org/2023/05/02/bio-formats-6-13-0.html.

Would it be worth to revert test120 to OMERO.server 5.6.7 to confirm whether the issue is related to this dependency upgrade ?

will-moore commented 1 year ago

@sbesson yes, if that's the easiest option to get this working, and we don't need the 5.6.8, that would be great

will-moore commented 1 year ago

Thanks @sbesson that's working now. After updating ZarrReader again on the 4 servers I can see NGFF images.

NB: I noticed a couple of warnings when restarting servers on omeroreadonly-3 and omeroreadonly-4. Not sure what that means?

[wmoore@test120-omeroreadonly-3 ~]$ sudo service omero-server restart
Redirecting to /bin/systemctl restart omero-server.service
Message from syslogd@localhost at Sep  7 15:02:01 ...
 haproxy[18172]: backend omero4064-1 has no server available!

sbesson commented 1 year ago

That's the HAproxy application which is balancing the load between read-only servers and responding to the service getting restarted. If you look at /var/log/messages under the proxy server, you will see that it the connection is recovered eventually

Sep  7 15:02:01 localhost haproxy[18172]: Server omero4064-1/omero4064-192_168_4_90 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:02:01 localhost haproxy[18172]: backend omero4064-1 has no server available!
Sep  7 15:02:02 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_90 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:02:17 localhost haproxy[18172]: Server omero4064-1/omero4064-192_168_4_90 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:02:18 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_90 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:03:09 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_68 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:03:10 localhost haproxy[18172]: Server omero4064-0/omero4064-192_168_4_68 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:03:10 localhost haproxy[18172]: backend omero4064-0 has no server available!
Sep  7 15:03:26 localhost haproxy[18172]: Server omero4064-0/omero4064-192_168_4_68 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:03:27 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_68 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in que

will-moore commented 1 year ago

Going to test a plate on idr-testing... idr0125...

  idr0025/10x images plate 3.ome.zarr,S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea,23152
  idr0025/10x images plate 1.ome.zarr,S-BIAD846/52304cdf-4eba-4f0a-84b1-690e0d66add9,23151
  idr0025/10x images plate 2.ome.zarr,S-BIAD846/72cc291b-a4e0-4807-bd23-22e9ad75c0dd,23201

oops - forgot to remove whitespace from the csv above, but it still seems to find the right strings eventually...

$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
> done
bash: idr0025/10x.sql: No such file or directory
usage: /opt/omero/server/venv3/bin/omero mkngff sql [-h] [--secret SECRET]
                                                    [--zarr_name ZARR_NAME]
                                                    [--symlink_repo SYMLINK_REPO]
                                                    fileset_id symlink_target
/opt/omero/server/venv3/bin/omero mkngff sql: error: argument fileset_id: invalid int value: 'images'
usage: /opt/omero/server/venv3/bin/omero mkngff sql [-h] [--secret SECRET]
                                                    [--zarr_name ZARR_NAME]
                                                    [--symlink_repo SYMLINK_REPO]
                                                    fileset_id symlink_target
/opt/omero/server/venv3/bin/omero mkngff sql: error: argument fileset_id: invalid int value: 'plate'
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/13 // 15-26-46.818 for fileset 23152
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr -> /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr
...

will-moore commented 1 year ago

$ for r in $(cat $IDRID.csv); do
>   fsid=$(echo $r | cut -d',' -f3)
>   psql -U omero -d idr -h $DBHOST -f "$fsid.sql"
> done
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311585
(1 row)

COMMIT
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311586
(1 row)

COMMIT
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311587
(1 row)

COMMIT

Viewed first plate image (to start memo file generation - 11:42)...

will-moore commented 1 year ago

ON http://localhost:1080/webclient/ got this while waiting on memo file....

DatabaseBusyException
 message = cannot create transaction

https://idr-testing.openmicroscopy.org/webclient/?show=image-5025551 now loads right panel but image doesn't display.

https://idr-testing.openmicroscopy.org/webclient/imgData/3260451/ takes time and fails with 504 Gateway Time-out.

# on readonly-1

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "mkngff"

2023-09-07 15:08:44,105 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr/OME/.METADATA.ome.xml.bfmemo (46996 bytes)
2023-09-07 15:08:44,105 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1694099317085] time[7019] tag[loci.formats.Memoizer.setId]
2023-09-07 15:08:44,111 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-07 19:41:25,964 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-01/30/17-28-12.832/.front.companion.ome.bfmemo (1616957 bytes)

restart all servers... and web servers...

ssh -A idr-testing.openmicroscopy.org

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-web restart"; done

But still see Gateway Timeout at https://idr-testing.openmicroscopy.org/webclient/imgData/3260451/

will-moore commented 1 year ago

Going to update ZarrReader to hopefully improve performance...

for each of omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4...

ssh omeroreadwrite
sudo -u omero-server -s
cd
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/

Then restarted all 5 servers in loop as above.

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

will-moore commented 1 year ago

Now try set-id on the 2nd Plate of idr0025 above (only tried Plate 1 so far)... We want to specify which server to test with (instead of letting nginx choose with ip-hash)

ssh idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80

First checked Plate 1 image above http://localhost:1080/webclient/?show=image-3260451 and this was viewable! Checked memo time, using prefix for this Fileset "590_mkngff"...

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "590_mkngff"
2023-09-11 13:21:00,608 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-11 13:21:00,608 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694438408707] time[51900] tag[loci.formats.Memoizer.setId]
2023-09-11 13:21:00,608 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml Series: 0

However - this isn't the original memo file regeneration, which I attempted to start this morning. Couldn't see logs for that on any of the 5 servers.

Now tried viewing image on Plate 2...

will-moore commented 1 year ago

Image from Plate 2 of idr0025 is now "viewable" (Preview panel loads in webclient)...

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "747_mkngff"
2023-09-11 15:06:01,615 DEBUG [                   loci.formats.Memoizer] (l.Server-8) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-11 15:06:01,615 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1694439010614] time[5751000] tag[loci.formats.Memoizer.setId]
2023-09-11 15:06:01,615 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 0

5751000 ms is 95 minutes.

Unfortunately the Images are rendering black:

https://idr-testing.openmicroscopy.org/webclient/?show=image-3261251

will-moore commented 1 year ago

Testing with today's merge OMEZarrReader.jar on the 3rd idr0025 Plate

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "46.818_mkngff"
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694516674666] time[244964] tag[loci.formats.Memoizer.setId]
2023-09-12 11:08:39,631 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0

244964 ms is 4 minutes

sbesson commented 1 year ago

As discussed during the weekly IDR calls, the following commands aim to measure the initialization time for several of the IDR datasets converted to NGFF in order to benchmark/assess the current performance of ZarrReader against the original reader (ScreenReader for many of these). These measurements use the Bio-Formats command-line utility downloaded from the OME website with the bioformats_package JAR

idr0016

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/20607.screen | grep Init
Initializing reader
Initialization took 1136.254s
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/20607.screen | grep Init
Initializing reader
Initialization took 130.914s

idr0033

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0033-rohban-pathways/screens/41744.screen  | grep Init
Initializing reader
Initialization took 1145.868s                                                                                                                                                                                                                    
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0033-rohban-pathways/screens/41744.screen  | grep Init
Initializing reader
Initialization took 318.314s

idr0025

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix  /uod/idr/metadata/idr0025-stadler-proteinatlas/screens/10x\ images\ plate\ 2.screen | grep Init
Initializing reader
Initialization took 70.219s
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix  /uod/idr/metadata/idr0025-stadler-proteinatlas/screens/10x\ images\ plate\ 2.screen | grep Init
Initializing reader
Initialization took 5.566s

Overall, these timings should be used for an informal comparison with the initialization times described in the issue description. Note a few comments:

by design, the commands above only measure the reader initialization time which I expect to be the bulk of the memo file regeneration. If necessary, we could look into a more involved testing workflow by deleting the memo files in a pilot with the current software and regenerating them using the same approach as the one above
by design, the commands above only measure the reader initialization time which I expect to be the bulk of the memo file regeneration. If necessary, we could look into a more involved testing workflow by deleting the memo files in a pilot with the current software and regenerating them using the same approach as the one above
Bio-Formats showinf utility is invoked a few times successfully. The first measurement effectively reflects the initialization tme without any caching while the subsequent ones benefit from some of the filesystem level caching associated with the first command. Freeing the cache using https://linux-mm.org/Drop_Caches allows to partly restore the initialization time associated with the first invocation.
For a faithful comparison, it would be useful to know underI don't know under which regime the ZarrReader memo file regeneration might be working

will-moore commented 1 year ago

Comparing the first initialisation time for pattern files above with the times to create bfmemo for equivalent NGFF filesets (from description above):

idr0016 - pattern: 19 mins, NGFF: 7.5 mins
idr0033 - pattern: 19 mins, NGFF: 81 mins
idr0025 - pattern: 1.2 mins, NGFF: 4 - 12 mins

So, NGFF is ~ 2x quicker for idr0016 but ~ 4x slower for the others.

joshmoore commented 1 year ago

:+1: for the data but still concerning.

IDR / idr-metadata

Testing mkngff on idr-testing #672