search

IDR / idr0125-way-cellpainting

0 stars 1 forks source link

idr0125 NGFF import #4

Open will-moore opened 10 months ago

will-moore commented 10 months ago

On idr0125-pilot, going to test import of METADATA.ome.xml and then use mkngff to replace the fileset with full NGFF plate mounted on s3.

sudo cp idr0125/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr/OME/METADATA.ome.xml /opt/omero/server/
import as omero-server user...

omero import --transfer=ln_s --skip=all METADATA.ome.xml --file /tmp/idr0125_xml.log  --errs /tmp/idr0125_xml.err
$ tail -f /tmp/idr0125_xml.err
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-09-04 16:00:07,994 15499      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-09-04 16:00:31,170 38675      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-09-04 16:00:31,188 38693      [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /opt/omero/server/idr0125/METADATA.ome.xml...
2023-09-04 16:00:31,412 38917      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,219 39724      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,391 39896      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-09-04 16:00:32,673 40178      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 52201342

Other imported objects:
Fileset:5287519

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3447 images imported, 0 errors in 0:59:35.161

Running mkngff to replace OME.METADATA.ome.xml single-file Fileset with NGFF plate...

$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287519 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr > 5287519.sql

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04 // 16-00-19.149 for fileset 5287519
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr

$ psql -U omero -d idr -h $DBHOST -f 5287519.sql 
UPDATE 3447
BEGIN
 mkngff_fileset 
----------------
        5287569
(1 row)

COMMIT
will-moore commented 10 months ago

Updated bulk import yaml to import METADATA.ome.xml for every plate...

I went run bulk import on idr0125-pilot...

omero import --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20230918.log  --errs /tmp/idr0125_20230918.err

but found that this failed because every Plate.zarr directory is empty.

ls /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/

Could this be due to the update of symlinks in ManagedRepo? Commands are at https://github.com/IDR/idr0125-way-cellpainting/blob/d883ec3a88178b3aaf207acbf9612ef0e5e3f840/scripts/symlinks.bash

E.g.

cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-83/2022-12/14/10-51-00.924
rm -rf SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
ln -s /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

But this shouldn't affect what's under /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/ ??

will-moore commented 10 months ago

Instead, lets simply download and import the OME/METADATA.ome.xml files...

NB: this loop repeats for every - in each yaml row as well as the actual plate names! 

for r in $(cat /uod/idr/metadata/idr0125-way-cellpainting/docs/_data/plates.yml); do
  cd
  cd idr0125
  echo $r
  mkdir -p $r
  cd $r
  wget https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$r/OME/METADATA.ome.xml
done
will-moore commented 10 months ago

Needed to rename each dir to remove .ome.zarr because trying to import e.g. SQ00015117__2016-04-13T18_12_45-Measurement1.ome.zarr/METADATA.ome.xml failed looking for .zgroup etc.

for i in $(ls ./); do
  mv $i ${i/.ome.zarr/}
done
for i in $(ls ./); do
  omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log  --errs /tmp/$i.err
done
will-moore commented 10 months ago 
$ tail -f /tmp/SQ00014812__2016-05-23T20_44_31-Measurement1.err

2023-09-18 19:01:25,306 3563454    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 52313944
2023-09-18 19:01:26,455 3564603    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1/METADATA.ome.xml
Other imported objects:
Fileset:5287672

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 0:59:19.730
will-moore commented 9 months ago

Seems like import got stopped at some point (maybe due to server restart)... Last log files are:

-rw-r--r--.  1 omero-server omero-server 4.3K Sep 19 12:33 SQ00015050__2016-05-24T06_38_27-Measurement1.err
-rw-r--r--.  1 omero-server omero-server    0 Sep 19 11:54 SQ00015050__2016-05-24T06_38_27-Measurement1.log

Moved the imported dirs out of idr0125/ into idr00125_imported/

ls idr0125_imported/
SQ00014812__2016-05-23T20_44_31-Measurement1  SQ00014816__2016-05-23T14_07_55-Measurement1  SQ00014820__2016-05-25T21_16_25-Measurement1  SQ00015044__2016-05-20T05_34_55-Measurement1  SQ00015048__2016-05-25T22_53_31-Measurement1
SQ00014813__2016-05-23T19_03_28-Measurement1  SQ00014817__2016-05-23T12_08_18-Measurement1  SQ00015041__2016-05-25T19_39_25-Measurement1  SQ00015045__2016-05-20T03_57_42-Measurement1  SQ00015049__2016-05-24T08_17_26-Measurement1
SQ00014814__2016-05-23T17_24_56-Measurement1  SQ00014818__2016-05-23T10_30_25-Measurement1  SQ00015042__2016-05-25T18_02_08-Measurement1  SQ00015046__2016-05-20T02_20_03-Measurement1  SQ00015050__2016-05-24T06_38_27-Measurement1
SQ00014815__2016-05-23T15_46_08-Measurement1  SQ00014819__2016-05-23T08_51_51-Measurement1  SQ00015043__2016-05-25T16_25_21-Measurement1  SQ00015047__2016-05-20T00_41_45-Measurement1

and restarted import in for loop as above, from idr0125/

will-moore commented 9 months ago

Moved SQ00015110__2016-05-05T04_50_06-Measurement1 and other Plates before it into idr0125_imported and restarted import from idr0125 as above...

for i in $(ls ./); do   omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log  --errs /tmp/$i.err
will-moore commented 9 months ago

For running mkngff on idr0125-pilot, we have data mounted at

$ ls /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/

Lets run mkngff on this Plate/Filesets... as regular wmoore user on idr0125-pilot...

Screenshot 2023-09-28 at 15 28 08

(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ omero mkngff sql --secret=$SECRET 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr > idr0125_sql/5287672.sql
Using session for public@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024 for fileset: 5287672
...
$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287672.sql 
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5288197
(1 row)
COMMIT

as omero-server...

(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Viewing... http://localhost:1040/webclient/?show=well-2339987

EDIT: after 3 days this image(s) are still not viewable. Probably just takes too long for ZarrReader to read a BIG plate with all the data being remote in USA.

will-moore commented 9 months ago

All METADATA.ome.xml imported above. Last batch - first plate SQ00015111__2016-05-05T03_10_03-Measurement1 created at 2023-09-28 15:02:03 Last plate imported (created) at 2023-10-02 10:23:45.

94 Plates imported in about 90 hours. 1 hour per plate!

will-moore commented 9 months ago

Since running mkngff above, and attempting to view image at http://localhost:1040/webclient/?show=well-2339987 (which still isn't visible) we now have ZarrReader update on idr0125-pilot.

Try another plate... 2nd plate: SQ00014813__2016-05-23T19_03_28-Measurement1

Fileset ID: 5287673

As wmoore user on idr0125-pilot...

omero mkngff sql --secret=$SECRET 5287673 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr > idr0125_sql/5287673.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585 for fileset: 5287673
...
(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287673.sql
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5288270
(1 row)

COMMIT

As omero-server user...

(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287673 "/cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr

View image at http://localhost:1040/webclient/?show=image-14847759 ...

will-moore commented 9 months ago

This image is also still not viewable. I think we have to conclude that this approach isn't going to work. There's probably just too many files to load remotely (from the US).

will-moore commented 9 months ago

Since the size of a "metadata-only" plate appears to be moderate, we can look again at the workflow tested earlier:

(base) [wmoore@pilot-idr0125-omeroreadwrite idr0125]$ du -sh ./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/
82M ./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/
will-moore commented 9 months ago

With idr0125_plates.csv:

SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr
SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr
SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr
SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr
SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr
SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr
SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr
SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr
SQ00015041__2016-05-25T19_39_25-Measurement1.ome.zarr
SQ00015042__2016-05-25T18_02_08-Measurement1.ome.zarr
SQ00015043__2016-05-25T16_25_21-Measurement1.ome.zarr
SQ00015044__2016-05-20T05_34_55-Measurement1.ome.zarr
SQ00015045__2016-05-20T03_57_42-Measurement1.ome.zarr
SQ00015046__2016-05-20T02_20_03-Measurement1.ome.zarr
SQ00015047__2016-05-20T00_41_45-Measurement1.ome.zarr
SQ00015048__2016-05-25T22_53_31-Measurement1.ome.zarr
SQ00015049__2016-05-24T08_17_26-Measurement1.ome.zarr
SQ00015050__2016-05-24T06_38_27-Measurement1.ome.zarr
SQ00015051__2016-05-24T05_00_04-Measurement1.ome.zarr
SQ00015052__2016-05-24T03_21_49-Measurement1.ome.zarr
SQ00015053__2016-05-24T01_42_05-Measurement1.ome.zarr
SQ00015054__2016-05-24T00_02_16-Measurement1.ome.zarr
SQ00015055__2016-05-23T22_23_05-Measurement1.ome.zarr
SQ00015056__2016-05-20T16_27_19-Measurement1.ome.zarr
SQ00015057__2016-05-20T14_48_37-Measurement1.ome.zarr
SQ00015058__2016-05-20T08_50_58-Measurement1.ome.zarr
SQ00015059__2016-05-20T07_12_47-Measurement1.ome.zarr
SQ00015096__2016-06-08T17_05_23-Measurement1.ome.zarr
SQ00015097__2016-06-08T15_26_27-Measurement1.ome.zarr
SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr
SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr
SQ00015100__2016-05-16T22_01_19-Measurement1.ome.zarr
SQ00015101__2016-05-16T20_23_02-Measurement1.ome.zarr
SQ00015102__2016-05-16T18_45_00-Measurement1.ome.zarr
SQ00015103__2016-05-16T17_06_53-Measurement1.ome.zarr
SQ00015105__2016-05-06T00_10_33-Measurement1.ome.zarr
SQ00015106__2016-05-05T22_30_28-Measurement1.ome.zarr
SQ00015107__2016-05-05T09_50_51-Measurement1.ome.zarr
SQ00015108__2016-05-05T08_10_09-Measurement1.ome.zarr
SQ00015109__2016-05-05T06_29_59-Measurement1.ome.zarr
SQ00015110__2016-05-05T04_50_06-Measurement1.ome.zarr
SQ00015111__2016-05-05T03_10_03-Measurement1.ome.zarr
SQ00015112__2016-05-05T01_29_08-Measurement1.ome.zarr
SQ00015116__2016-04-13T16_31_49-Measurement1.ome.zarr
SQ00015117__2016-04-13T18_12_45-Measurement1.ome.zarr
SQ00015118__2016-04-13T19_52_28-Measurement1.ome.zarr
SQ00015119__2016-04-13T21_33_42-Measurement1.ome.zarr
SQ00015120__2016-04-13T23_14_32-Measurement1.ome.zarr
SQ00015121__2016-04-14T00_54_07-Measurement1.ome.zarr
SQ00015122__2016-04-14T02_33_42-Measurement1.ome.zarr
SQ00015123__2016-04-14T04_14_45-Measurement1.ome.zarr
SQ00015124__2016-04-14T07_35_09-Measurement1.ome.zarr
SQ00015125__2016-04-13T14_53_08-Measurement2.ome.zarr
SQ00015126__2016-03-25T14_15_31-Measurement1.ome.zarr
SQ00015127__2016-04-14T05_53_59-Measurement1.ome.zarr
SQ00015128__2016-04-14T21_08_47-Measurement1.ome.zarr
SQ00015129__2016-04-15T05_30_25-Measurement1.ome.zarr
SQ00015130__2016-04-01T13_10_04-Measurement1.ome.zarr
SQ00015131__2016-03-25T17_23_46-Measurement1.ome.zarr
SQ00015132__2016-04-24T07_20_38-Measurement1.ome.zarr
SQ00015133__2016-04-24T08_59_07-Measurement1.ome.zarr
SQ00015134__2016-06-12T00_30_52-Measurement1.ome.zarr
SQ00015135__2016-06-11T22_53_05-Measurement1.ome.zarr
SQ00015136__2016-06-11T21_14_31-Measurement1.ome.zarr
SQ00015137__2016-06-11T19_36_55-Measurement1.ome.zarr
SQ00015138__2016-06-11T17_59_08-Measurement1.ome.zarr
SQ00015139__2016-06-11T16_21_04-Measurement1.ome.zarr
SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr
SQ00015141__2016-06-09T08_43_58-Measurement1.ome.zarr
SQ00015142__2016-05-19T23_03_33-Measurement1.ome.zarr
SQ00015143__2016-05-19T21_25_23-Measurement1.ome.zarr
SQ00015144__2016-05-19T19_47_38-Measurement1.ome.zarr
SQ00015145__2016-05-19T18_10_43-Measurement1.ome.zarr
SQ00015146__2016-05-19T10_37_52-Measurement1.ome.zarr
SQ00015147__2016-05-17T07_49_53-Measurement1.ome.zarr
SQ00015148__2016-05-17T01_17_02-Measurement1.ome.zarr
SQ00015149__2016-05-16T23_39_00-Measurement1.ome.zarr
SQ00015150__2016-06-09T06_45_35-Measurement1.ome.zarr
SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr
SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr
SQ00015153__2016-06-09T01_50_39-Measurement1.ome.zarr
SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr
SQ00015155__2016-04-14T17_48_56-Measurement1.ome.zarr
SQ00015156__2016-04-14T19_29_39-Measurement1.ome.zarr
SQ00015157__2016-04-14T22_48_58-Measurement1.ome.zarr
SQ00015158__2016-04-15T00_30_51-Measurement1.ome.zarr
SQ00015159__2016-04-15T02_10_36-Measurement1.ome.zarr
SQ00015160__2016-04-15T03_50_42-Measurement1.ome.zarr
SQ00015162__2016-04-15T08_49_32-Measurement1.ome.zarr
SQ00015163__2016-04-20T21_00_28-Measurement1.ome.zarr
SQ00015164__2016-04-20T22_38_54-Measurement1.ome.zarr
SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr
SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr
SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr
SQ00015168__2016-04-21T05_12_27-Measurement1.ome.zarr
SQ00015169__2016-04-21T06_50_28-Measurement1.ome.zarr
SQ00015170__2016-04-21T08_29_08-Measurement1.ome.zarr
SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr
SQ00015172__2016-04-21T18_01_12-Measurement1.ome.zarr
SQ00015173__2016-04-15T07_09_23-Measurement1.ome.zarr
SQ00015194__2016-03-25T12_39_04-Measurement1.ome.zarr
SQ00015195__2016-04-24T21_41_05-Measurement2.ome.zarr
SQ00015196__2016-04-24T23_20_24-Measurement2.ome.zarr
SQ00015197__2016-04-24T13_30_35-Measurement1.ome.zarr
SQ00015198__2016-04-24T15_10_23-Measurement1.ome.zarr
SQ00015199__2016-04-23T15_43_06-Measurement1.ome.zarr
SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr
SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr
SQ00015202__2016-04-23T22_17_34-Measurement1.ome.zarr
SQ00015203__2016-04-25T00_59_49-Measurement2.ome.zarr
SQ00015204__2016-04-24T20_02_01-Measurement2.ome.zarr
SQ00015205__2016-04-24T02_21_50-Measurement1.ome.zarr
SQ00015206__2016-04-24T04_00_27-Measurement1.ome.zarr
SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr
SQ00015208__2016-04-22T16_36_31-Measurement1.ome.zarr
SQ00015209__2016-04-22T18_16_23-Measurement1.ome.zarr
SQ00015210__2016-04-25T19_58_35-Measurement1.ome.zarr
SQ00015211__2016-04-23T17_22_06-Measurement1.ome.zarr
SQ00015212__2016-04-23T19_01_00-Measurement1.ome.zarr
SQ00015214__2016-05-04T23_48_48-Measurement1.ome.zarr
SQ00015215__2016-05-04T18_49_26-Measurement1.ome.zarr
SQ00015216__2016-05-04T17_08_56-Measurement1.ome.zarr
SQ00015217__2016-05-04T22_08_57-Measurement1.ome.zarr
SQ00015218__2016-05-04T08_36_42-Measurement1.ome.zarr
SQ00015219__2016-05-04T15_29_24-Measurement1.ome.zarr
SQ00015220__2016-05-04T20_29_20-Measurement1.ome.zarr
SQ00015221__2016-05-04T06_58_09-Measurement1.ome.zarr
SQ00015222__2016-05-06T05_09_24-Measurement1.ome.zarr
SQ00015223__2016-05-06T03_29_13-Measurement1.ome.zarr
SQ00015224__2016-05-06T01_49_03-Measurement1.ome.zarr
SQ00015229__2016-05-13T08_10_01-Measurement1.ome.zarr
SQ00015230__2016-05-12T18_00_11-Measurement1.ome.zarr
SQ00015231__2016-05-12T09_02_26-Measurement1.ome.zarr
SQ00015232__2016-05-12T07_24_32-Measurement1.ome.zarr
SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr
(base) [wmoore@pilot-idr0125-omeroreadwrite ngff]$ df -h /ngff
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   87G   14G  87% /
cd /ngff
for zarr in $(cat idr0125_plates.csv);
  do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done
...
11:21am...

...completed 18:54. 7 hrs 33 mins to download all metadata-only plates.

Used 9 GB of space, based on what's left:

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ df -h /ngff
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   96G  4.8G  96% /
will-moore commented 9 months ago

Started import the metadata-only plates downloaded to idr0125-pilot at /ngff/idr0125 using bulk.yml from https://github.com/IDR/idr0125-way-cellpainting/commit/8973828b3959e2e35b38acdbeec32c886bd91957

screen -r 17598.idr0125_import
...
(venv3) (base) bash-4.2$ omero import --depth=100 --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231011.log --errs /tmp/idr0125_20231011.err
...
2023-10-11 11:49:19,024 197526     [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-10-11 11:49:22,703 201205     [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-10-11 11:49:23,653 202155     [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-10-11 11:49:27,030 205532     [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
2023-10-11 11:49:27,307 205809     [      main] ERROR                loci.formats.FormatHandler - ZarrReader returning list of used files of size: 11157
2023-10-11 11:49:28,816 207318     [      main] INFO      ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 1 group(s) with 1 call(s) to setId in 171492ms. (197094ms total) [0 unknowns]
2023-10-11 11:49:29,672 208174     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-11 11:49:31,602 210104     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-11 11:49:33,942 212444     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-11 11:49:33,958 212460     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-11 11:49:33,958 212460     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-10-11 11:49:33,958 212460     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-10-11 11:49:33,959 212461     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-11 11:49:33,959 212461     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-11 11:49:33,959 212461     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-10-11 11:49:34,669 213171     [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Screen:name:idr0125-way-cellpainting/screenA
2023-10-11 11:49:34,887 213389     [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Screen:3351
...
will-moore commented 9 months ago

51 plates imported so far: ~ 10 per day.

will-moore commented 9 months ago

Test import of first Plate on idr-testing with ZarrReader: /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_p643_b565.jar from https://github.com/IDR/idr-metadata/issues/675#issuecomment-1755198551

Using aws as wmoore in aws conda env...

$ aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Moved data to /ngff/idr0125/ and chown to omero-server. Import as omero-server...

$ omero import --depth=100 --transfer=ln_s --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125.log  --errs /tmp/idr0125.err
2023-10-17 10:57:54,260 291        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2023-10-17 10:57:54,280 311        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2023-10-17 10:57:54,344 375        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-10-17 10:57:54,345 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-10-17 10:57:54,345 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-10-17 10:57:54,345 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-10-17 10:57:54,345 376        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-10-17 10:57:54,348 379        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-17 10:57:54,716 747        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-10-17 10:58:03,322 9353       [      main] ERROR                loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-10-17 10:58:03,322 9353       [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
2023-10-17 10:58:03,329 9360       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/0/.zarray
2023-10-17 10:58:03,330 9361       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zgroup
2023-10-17 10:58:03,330 9361       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zattrs

...

2023-10-17 10:58:03,653 9684       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zattrs
2023-10-17 10:58:03,653 9684       [      main] ERROR                loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/.zgroup
2023-10-17 10:58:04,482 10513      [      main] INFO      ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 0 group(s) with 1 call(s) to setId in 8723ms. (9766ms total) [0 unknowns]
2023-10-17 10:58:04,530 10561      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-17 10:58:05,062 11093      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-17 10:58:05,450 11481      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-17 10:58:05,464 11495      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.99.1.el7.x86_64
No imports found
will-moore commented 9 months ago

Also on idr0125-pilot, wanted to test whether import with chunks is working with current ZarrReader (which is likely 0.3.1 and Canonical paths PR)... EDIT - probably this is Performace PR before getUsedFiles - this works for import and for viewing mkngff Filesets after memo is generated - but doesn't work for generating memo file after mkngff.

Import idr0054 data directly from goofys mount of BIA bucket...

(venv3) (base) bash-4.2$ omero import -d 15301 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054.log  --errs /tmp/idr0054.err

Or from local copy...

omero import -d 15301 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054a.log  --errs /tmp/idr0054a.err
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-10-17 12:39:08,698 300        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-10-17 12:39:08,717 319        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: e4b0b24f6f1cc9b98ea7edc03f218a631d79d0d2 date: 20230925-0010
2023-10-17 12:39:08,782 384        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-17 12:39:09,251 853        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-10-17 12:39:09,275 877        [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-17 12:39:09,280 882        [      main] ERROR                loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,286 888        [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-10-17 12:39:09,679 1281       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,694 1296       [      main] ERROR                loci.formats.FormatHandler - ZarrReader generating group keys
2023-10-17 12:39:09,694 1296       [      main] ERROR                loci.formats.FormatHandler - ZarrReader adding group keys from ZarrService
2023-10-17 12:39:09,729 1331       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-10-17 12:39:09,736 1338       [      main] ERROR                loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-10-17 12:39:09,736 1338       [      main] ERROR                loci.formats.FormatHandler - ZarrReader adding Array Keys from ZarrService
2023-10-17 12:39:09,749 1351       [      main] ERROR                loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-10-17 12:39:09,749 1351       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/0
2023-10-17 12:39:09,749 1351       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,764 1366       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,765 1367       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/1
2023-10-17 12:39:09,765 1367       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1
2023-10-17 12:39:09,766 1368       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,766 1368       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/2
2023-10-17 12:39:09,766 1368       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,767 1369       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/2
2023-10-17 12:39:09,767 1369       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,767 1369       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/3
2023-10-17 12:39:09,768 1370       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,768 1370       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/3
2023-10-17 12:39:09,769 1371       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,769 1371       [      main] ERROR                loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/4
2023-10-17 12:39:09,769 1371       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,770 1372       [      main] ERROR                loci.formats.FormatHandler - Opening zarr for series 4 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4
2023-10-17 12:39:09,770 1372       [      main] ERROR                loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,771 1373       [      main] ERROR                loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-10-17 12:39:09,771 1373       [      main] ERROR                loci.formats.FormatHandler - ZarrReader parsing plate
2023-10-17 12:39:09,771 1373       [      main] ERROR                loci.formats.FormatHandler - ZarrReader initialization complete
2023-10-17 12:39:09,772 1374       [      main] ERROR                loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,790 1392       [      main] ERROR                loci.formats.FormatHandler - ZarrReader returning list of used files of size: 444
2023-10-17 12:39:09,835 1437       [      main] INFO      ome.formats.importer.ImportCandidates - 444 file(s) parsed into 1 group(s) with 1 call(s) to setId in 540ms. (584ms total) [0 unknowns]
2023-10-17 12:39:09,884 1486       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-17 12:39:10,415 2017       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-17 12:39:10,744 2346       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-17 12:39:10,756 2358       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-17 12:39:10,756 2358       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-10-17 12:39:10,756 2358       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-10-17 12:39:10,756 2358       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-17 12:39:10,757 2359       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-17 12:39:10,757 2359       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-10-17 12:39:11,315 2917       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:15301
2023-10-17 12:39:11,660 3262       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:15301
2023-10-17 12:39:11,670 3272       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-10-17 12:39:34,326 25928      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-10-17 12:39:34,459 26061      [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml...
2023-10-17 12:39:34,503 26105      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-17 12:39:34,735 26337      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml

...

2023-10-17 12:42:18,693 190295     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4/0/9/0/0/0
2023-10-17 12:42:22,352 193954     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-17 12:42:22,477 194079     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 53840675
2023-10-17 12:42:24,570 196172     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 53840675
2023-10-17 12:42:34,559 206161     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 53840675
2023-10-17 12:42:36,168 207770     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 53840675
2023-10-17 12:42:36,252 207854     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 53840675
2023-10-17 12:42:36,463 208065     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 53840675
2023-10-17 12:42:37,860 209462     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5288333

==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:03:26.856

Screenshot 2023-10-17 at 13 44 53

will-moore commented 9 months ago

Since import from https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1757512153 got interrupted (server restarts above), 61 plates were imported, so deleted 61 rows from idr0125-screenA-bulk.yml - then an extra row since that was SQ00015133__2016-04-24T08_59_07-Measurement1 which was the last plate imported. Then restarted import...

(venv3) (base) bash-4.2$ omero import --depth=100 --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231018.log  --errs /tmp/idr0125_20231018.err

EDIT: oops - forgot to use --transfer=ln_s here!

will-moore commented 8 months ago

Importing metadata-only plates is still in progress on idr0125-pilot.

Testing symlinking etc... with https://github.com/IDR/idr-utils/pull/54

Moved some imported Plates into temp Screen:3352 to use for symlinking... As omero-server user...

$ cd /uod/idr/metadata
$ python idr-utils/scripts/managed_repo_symlinks.py Screen:3352 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

fileset_dirs {}

Fileset: 5288271 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-29/2023-10/11/11-57-46.608/
Render Image 15325028
fs_contents ['ngff']
Symlink target not found: /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/ngff

Fileset: 5288272 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/
Render Image 15331556
^C

A couple of issues:

will-moore commented 8 months ago

The 2nd issue (bullet point) above seems to be due to the path in plates.tsv, e.g. /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr https://github.com/IDR/idr0125-way-cellpainting/blob/8973828b3959e2e35b38acdbeec32c886bd91957/screenA/idr0125-screenA-plates.tsv

However, the usual behaviour elsewhere is for the path/to/plate.zarr truncated to just give the plate.zarr, which is added to the Fileset prefix. managed_repo/fileset_prefix/plate.zarr. This is what was happening for the first import...

This is probably because I didn't use --transfer=ln_s for this import...

Stopped import (restarted the server and updated ZarrReader to use OMEZarrReader-0.3.1-jar-with-dependencies.jar).

77 plates imported now. Last one was SQ00015150__2016-06-09T06_45_35-Measurement1. Updated sudo vi idr0125-screenA-plates.tsv to remove 17 lines upto and including that plate.

Restart...

omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231020.log  --errs /tmp/idr0125_20231020.err
will-moore commented 8 months ago

For the first 77 plates imported, we need to use a TEMP commit of https://github.com/IDR/idr-utils/pull/54/commits/2888b21c2751421ecf24a845c0b0da2aa6662184 to add in the /ngff/idr0125/ to the filesetPrefix... With this installed....

python idr-utils/scripts/managed_repo_symlinks.py Screen:3352 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
...

Fileset: 5288272 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/ngff/idr0125
Render Image 15331556
fs_contents ['SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/ngff/idr0125/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr

Fileset: 5288273 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2023-10/11/16-38-30.825/ngff/idr0125
Render Image 15338049
fs_contents ['SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2023-10/11/16-38-30.825/ngff/idr0125/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr
will-moore commented 8 months ago

Something strange going on.....

ALL 3 of the images rendered above (one for the each fileset) are orphans! E.g http://localhost:1040/iviewer/image_data/15325028/ They aren't found by /webclient/?show=image-15325028 because http://localhost:1040/webclient/api/paths_to_object/?image=15325028 returns "orphaned".

"meta": {
"imageName": "SQ00014813__2016-05-23T19_03_28-Measurement1 [P/24/0/0]",
"imageDescription": "",
"imageAuthor": "Public data",
"projectName": "Multiple",
"projectId": null,
"projectDescription": "",
"datasetName": "Multiple",
"datasetId": null,
"datasetDescription": "",
"wellSampleId": "",
"wellId": "",
"imageTimestamp": 1464050222,
"imageId": 15325028,
"pixelsType": "uint16"
},

Images are found in that script with:

    params.addId(fileset_id)
    params.page(0, 1)
    query = "select img from Image img where img.fileset.id=:id"
    first_image = conn.getQueryService().findAllByQuery(query, params, conn.SERVICE_OPTS)[0]
will-moore commented 8 months ago

It also appears that all plates imported above (without --transfer=ln_x give ResourceError when trying to view an Image from any field position except the first (index=0). This is true both before and after the symlink update.

will-moore commented 8 months ago

Seems that with the first 2 batches imported above (without in-place import) each Fileset creates lots of extra images: 6528 instead of 3456. 6528 / 384 is 17 images per Well, which would be 2 for every field except the first. NB: Every field except the first gives ResourceError when trying to view.

(venv3) (base) bash-4.2$ grep -A 1 "Summary" /tmp/idr0125_20231011.err
...
==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 6528 images imported, 0 errors in 2:10:24.594
--
==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 6528 images imported, 0 errors in 2:12:56.714
--
==> Summary
11157 files uploaded, 0 filesets created, 0 images imported, 0 errors in 1:19:27.922

Will need to re-import all those 77 plates!

will-moore commented 8 months ago

Checking last import logs /tmp/idr0125_20231020.err found a couple of Plates failed to import:

Edited idr0125-screenA-plates.tsv to try again with those 3 Plates, then with all others after SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr

screen -r idr0125_import and cancelled import. Restarted import... (still using OMEZarrReader-0.3.1-jar-with-dependencies.jar)...

(venv3) (base) bash-4.2$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231023.log  --errs /tmp/idr0125_20231023.err

EDIT... 3 failed imports re-tried above ALL failed with ClientPath match - I guess the partial import still leaves Filesets.

will-moore commented 8 months ago 
$ python idr-utils/scripts/managed_repo_symlinks.py Screen:3351 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

fileset_dirs {}

Fileset: 5288371 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2023-10/20/13-55-33.718/
Render Image 15829653
fs_contents ['SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2023-10/20/13-55-33.718/SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr

Fileset: 5288372 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-10/20/16-23-21.825/
Render Image 15831760
...

EDIT - Kinda slow to render Images and create symlinks: Took 75 mins, to do 17 Plates

will-moore commented 8 months ago

Render all images in 17 plates...

cd /uod/idr/metadata/idr0125-way-cellpainting/
screen -r idr0125_symlinks
source /opt/omero/server/venv3/bin/activate
omero login

for a in {10951..10967};
do
   echo "$a"
   omero --debug=2 render set Plate:$a screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_$i.log 2>/tmp/idr0125_renderplate_$i.err;
done

EDIT - after 75 mins...

$ cat /tmp/idr0125_renderplate_10967.err | wc
    573    2856   24626

$ tail /tmp/idr0125_renderplate_10967.err
...
Image:15830886 got thumbnail in 14.07s
Updated rendering settings for Image:15830869
Image:15830869 got thumbnail in 11.47s
Updated rendering settings for Image:15830870
Image:15830870 got thumbnail in 13.40s
Updated rendering settings for Image:15830871
Image:15830871 got thumbnail in 13.81s
Updated rendering settings for Image:15830872
Image:15830872 got thumbnail in 14.03s
Updated rendering settings for Image:15830873
Image:15830873 got thumbnail in 13.94s

EDIT - after nearly 5 hours

(base) [wmoore@pilot-idr0125-omeroreadwrite idr0125-way-cellpainting]$ cat /tmp/idr0125_renderplate_10967.err | wc
   2117   10576   90246

Approx 1000 thumbnails rendered (from plate of 3456 images). This is VERY slow, compared to https://idr-redmine.openmicroscopy.org/issues/228?issue_count=1&issue_position=1#note-110 (9 months ago) e.g.

Image:14609144 got thumbnail in 0.68s
Updated rendering settings for Image:14609127

Previously thumbnail generation was 20 x faster than now!! Whereas when we "SaveAll" in webclient, all the 384 thumbnails (for 1 Well index) are generated in 10-15 mins.

EDIT - after 20 hours - not getting any faster!

$ cat /tmp/idr0125_renderplate_10967.err | wc
   2310   11541   98459

$ tail -f /tmp/idr0125_renderplate_10967.err
...
Updated rendering settings for Image:15833469
Image:15833469 got thumbnail in 14.02s
Updated rendering settings for Image:15833470
Image:15833470 got thumbnail in 13.40s
Updated rendering settings for Image:15833453
will-moore commented 8 months ago

Decided to cancel import, since it may be slowing down the Thumbnail generation and I want to try different ZarrReader... Last Plate to be imported was SQ00015198__2016-04-24T15_10_23-Measurement1 10974.

Updated ZarrReader to use OMEZarrReader_performance_allfiles.jar from https://github.com/ome/ZarrReader/pull/64#issuecomment-1768612367 which works for NGFF Import. Restarting server will also kill the thumbnail generation above.

First, test Thumbnail creation on a new plate...

for i in {10953..10967}; do
    echo "$i"
    omero --debug=2 render set Plate:$i screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_$i.log 2>/tmp/idr0125_renderplate_$i.err;
done

This gives...

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-10/21/00-27-26.892/SQ00015156__2016-04-14T19_29_39-Measurement1.ome.zarr/OME/METADATA.ome.xml

I guess that Plate that was imported with older ZarrReader - then image rendered and symlinks updated before trying to view images with latest ZarrReader.jar.

will-moore commented 8 months ago

The OMEZarrReader_performance_allfiles.jar doesn't work for viewing Plates imported with OMEZarrReader-0.3.1-jar-with-dependencies.jar but let's at least test whether it works for Import...

Removed 10 rows from idr0125-screenA-plates.tsv, including the last plate SQ00015198__2016-04-24T15_10_23-Measurement1 above, and restarted import...

$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231024.log  --errs /tmp/idr0125_20231024.err
will-moore commented 8 months ago

First plate imported with OMEZarrReader_performance_allfiles.jar: SQ00015200__2016-04-01T08_46_01-Measurement1 (11001), render an Image in webclient (blank), then update symlinks for these 2 new plates:

Strangely, these plates have /ngff/idr0125/ in the Fileset path, even though they were imported with in-place import! Used managed_repo_symlinks.py TEMP commit to handle this as above...

$ python idr-utils/scripts/managed_repo_symlinks.py Plate:11001 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

Fileset: 5288421 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/24/14-43-25.039/ngff/idr0125
Render Image 15916984
fs_contents ['SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/24/14-43-25.039/ngff/idr0125/SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr

$ python idr-utils/scripts/managed_repo_symlinks.py Plate:11002 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

Fileset: 5288422 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-10/24/17-12-30.722/ngff/idr0125
Render Image 15923706
fs_contents ['SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-10/24/17-12-30.722/ngff/idr0125/SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr
will-moore commented 8 months ago

Plate:11001 - after "Save to All" takes 20 mins to generate thumbnails for first Field of all Wells:

Screenshot 2023-10-25 at 07 35 05

Took 15 mins to repeat "Save to All" thumbnails for field 2.

For the next Plate, compare render thumbnail generation...

omero --debug=2 render set Plate:11002 screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_11002.log 2>/tmp/idr0125_renderplate_11002.err;
will-moore commented 8 months ago

In about 2.5 hours of render set Plate:11002 the logs show less than 400 thumbnails generated... This is 2.4 thumbnails per minute! Even though the logs say got thumbnail in 0.60s etc.

$ cat /tmp/idr0125_renderplate_11002.err | grep "got thumbnail" | wc
    384    1920   14635

$ tail -f /tmp/idr0125_renderplate_11002.err
Updated rendering settings for Image:15920249
Image:15920249 got thumbnail in 0.71s
Updated rendering settings for Image:15920250
Image:15920250 got thumbnail in 0.60s
Updated rendering settings for Image:15920251
Image:15920251 got thumbnail in 0.63s
Updated rendering settings for Image:15920252
Image:15920252 got thumbnail in 0.73s
Updated rendering settings for Image:15920253
Image:15920253 got thumbnail in 0.79s

Now running 5 hours...

$ cat /tmp/idr0125_renderplate_11002.err | grep "got thumbnail" | wc
    774    3870   29498

Cancel and try different approach...

will-moore commented 8 months ago

Use render copy...

omero --debug=2 render copy Image:15910651 Plate:11004 > /tmp/idr0125_renderplate_11004.log 2>/tmp/idr0125_renderplate_11004.err;

Completed in less than 5 mins: - mix of Error and success...

$ tail /tmp/idr0125_renderplate_11004.err 
Error: Image:15930288
Error: Image:15930289
Error: Image:15930297
Error: Image:15930298
Image:15930281 got thumbnail in 0.96s
Image:15930272 got thumbnail in 0.65s
Image:15930263 got thumbnail in 0.50s
Image:15930254 got thumbnail in 0.57s
Image:15930245 got thumbnail in 0.58s
Image:15930236 got thumbnail in 0.44s

# one row per image...
$ cat /tmp/idr0125_renderplate_11004.err | wc
   3457    8075   82259

# Seems we only got 1 success per Well!!
$ cat /tmp/idr0125_renderplate_11004.err | grep "got thumbnail" | wc
    384    1920   14592

Hmmm - plate maybe not ready... Same issue as above with only 1 Image per Well being visible

Try different plate... 14:55...

$ omero --debug=2 render copy Image:15910651 Plate:11002 > /tmp/idr0125_copyplate_11002.log 2>/tmp/idr0125_copyplate_11002.err;
will-moore commented 8 months ago

This is going much slower... After about 30 mins...

$ cat /tmp/idr0125_copyplate_11002.err | wc
   1029    2405   24592

And is still seeing most images are errors:

$ cat /tmp/idr0125_copyplate_11002.err | grep thumb | wc
    112     560    4336
will-moore commented 8 months ago

This plate took a lot longer to copy rdefs, probably because I'd not updated symlinks on Plate:11004 so it was fast. For Plate:11002, took ~90 mins

$ cat /tmp/idr0125_copyplate_11002.err | wc
   3457    8075   82611

Again, just 384 thumbs created:

$ cat /tmp/idr0125_copyplate_11002.err | grep "thumb" | wc
    384    1920   14944

For the "got thumbnail" images (first image in each Well), the rendering settings result in white images. Other images didn't get rendering settings applied, so they generate thumbs with default settings:

Screenshot 2023-10-25 at 16 37 35

will-moore commented 8 months ago

Trying to debug the render copy using single images...

Copy from first image of Plate:11001 - (see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1778743245) to the first image in Well above... 

(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15910651 Image:15919024
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Rendering settings successfully copied                               to 1 images.
Image:15919024 got thumbnail in 7.27s

This "works" as before when doing whole Plate, but image is still "white".

Copy from same image to Image at 2nd field in Well - or from 1st field in this Well to 2nd field - both FAIL

(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15910651 Image:15919025
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Error: Image:15919025
Rendering settings successfully copied                               to 0 images.
(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15919024 Image:15919025
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Error: Image:15919025
Rendering settings successfully copied                               to 0 images.

Copy from 2nd Field to 3rd Field - this "works" ok 

(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15919025 Image:15919026
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Rendering settings successfully copied                               to 1 images.
Image:15919026 got thumbnail in 3.05s

I notice that the Channel Names are different between 1st Field of each well: HOECHST 33342, Alexa 488, 488 long, Alexa 568, Alexa 647 and other 8 Fields: 0, 1, 2, 3, 4 (no names). This probably explains failure to copy settings...

Checking Channels for these Images... For the first Image in a Well, we have 5 different LogicalChannels (with labels):

idr=> select id, logicalchannel from Channel where pixels=(select id from Pixels where Image=15919024);
    id    | logicalchannel 
----------+----------------
 49552120 |       16077311
 49552119 |       16077310
 49552118 |       16077309
 49552117 |       16077308
 49552116 |       16077307
(5 rows)

But for the 2nd Image in a Well, all the LogicalChannels are the same:

idr=> select id, logicalchannel from Channel where pixels=(select id from Pixels where Image=15919025);
    id    | logicalchannel 
----------+----------------
 49552125 |       16077312
 49552124 |       16077312
 49552123 |       16077312
 49552122 |       16077312
 49552121 |       16077312
(5 rows)
will-moore commented 8 months ago

Set symlinks on newly imported Plates...

for i in {11003..11010}; do
    echo "$i"
    python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
will-moore commented 8 months ago

After updating to https://github.com/ome/omero-cli-render/pull/59/commits/35089403998581db74d26e2f107bdb5b622ae953

$ omero --debug=2 render copy Image:15910651 Plate:11003 > /tmp/idr0125_PR59_copyplate_11003.log 2>/tmp/idr0125_PR59_copyplate_11003.err
will-moore commented 8 months ago

Nearly 11 hours... 

$ tail /tmp/idr0125_PR59_copyplate_11003.err
..
Image:15925282 got thumbnail in 23.75s
Image:15925265 got thumbnail in 0.10s
Image:15925266 got thumbnail in 24.15s
Image:15925267 got thumbnail in 25.36s
Image:15925268 got thumbnail in 23.04s
Image:15925269 got thumbnail in 23.97s
Image:15925270 got thumbnail in 26.02s
Image:15925271 got thumbnail in 23.07s
Image:15925272 got thumbnail in 24.89s
Image:15925273 got thumbnail in 23.91s
Image:15925256 got thumbnail in 0.10s

Every 9th thumbnail is much faster. These are the First field images that are white (see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1779561095).

EDIT: still not done, but cancelling now...

$ cat /tmp/idr0125_PR59_copyplate_11003.err | wc
   2323   11625   90325

The settings have not been copied properly (at-all?). This is from first Field of A1 in Plate:11001 to the whole Plate:11003.

Previously, using the webclient, "Save to All" saves from an Image to the same Plate, then also updates Thumbnails for the same Field index as that Image, so this wouldn't show differences between Fields.

will-moore commented 8 months ago

Testing settings on a new Plate:11004... Save rendering settings to an Image at 2nd Field - then "Save to All" in webclient. This applies those settings to all images int the Plate except for First Field.

Try copying settings from First Image in A1 to the same Plate using...

$ screen -r idr0125_symlink2
$ omero --debug=2 render copy Image:15936763 Plate:11005 > /tmp/idr0125_PR59_copyplate_11005.log 2>/tmp/idr0125_PR59_copyplate_11005.err

EDIT: 3 hours - progress

$ cat /tmp/idr0125_PR59_copyplate_11005.err | wc
    420    2110   16422

But this is still not working. omero render copy doesn't apply settings to Plate, even on compatible images:

Screenshot 2023-10-26 at 16 19 34

will-moore commented 8 months ago

Since we have issues with OMEZarrReader_performance_allfiles.jar above, reverted to use OMEZarrReader-0.3.1-jar-with-dependencies.jar - restarted server

Removed 18 rows from plates.tsv to include last-imported plate SQ00015217__2016-05-04T22_08_57-Measurement1 and restarted import...

$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231027.log  --errs /tmp/idr0125_20231
will-moore commented 8 months ago

Trying to view images from Plate:10953 (not previously viewed since updating symlinks is failing:

2023-10-27 13:46:14,419 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-13) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml
2023-10-27 13:46:14,419 ERROR [                ome.io.nio.PixelsService] (.Server-13) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
    at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
    at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
    at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
    at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:520)
    at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:650)
    at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:1498)
    at ome.services.ThumbnailCtx$2.actOnOneGroup(ThumbnailCtx.java:407)
    at ome.services.PerGroupActor.actOnByGroup(PerGroupActor.java:92)
    at ome.services.ThumbnailCtx.createAndPrepareMissingRenderingSettings(ThumbnailCtx.java:411)
    at ome.services.ThumbnailBean.getThumbnailByLongestSideSet(ThumbnailBean.java:1003)
    at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source)
will-moore commented 8 months ago

Blitz logs don't look good just now - importing and trying to view too...

DatabaseBusyException etc ``` at ome.services.blitz.fire.Ring.checkPassword(Ring.java:270) at ome.services.blitz.fire.PermissionsVerifierI.checkPermissions(PermissionsVerifierI.java:97) at Glacier2._PermissionsVerifierDisp.___checkPermissions(_PermissionsVerifierDisp.java:114) at Glacier2._PermissionsVerifierDisp.__dispatch(_PermissionsVerifierDisp.java:149) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource 783e9342-628e-4aef-be4b-b32a8d81708e still empty after 30s wait time at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:319) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328) ... 25 common frames omitted 2023-10-27 14:08:30,039 ERROR [ o.s.blitz.fire.PermissionsVerifierI] (.Server-24) Exception thrown while checking password for:public org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy85.doWork(Unknown Source) at ome.services.util.Executor$Impl.executeSql(Executor.java:549) at ome.services.blitz.fire.Ring.checkPassword(Ring.java:270) at ome.services.blitz.fire.PermissionsVerifierI.checkPermissions(PermissionsVerifierI.java:97) at Glacier2._PermissionsVerifierDisp.___checkPermissions(_PermissionsVerifierDisp.java:114) at Glacier2._PermissionsVerifierDisp.__dispatch(_PermissionsVerifierDisp.java:149) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: ome.conditions.DatabaseBusyException: Cannot acquire connection at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:113) at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246) ... 20 common frames omitted 2023-10-27 14:08:30,596 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-15) Found location in errorTimes: -1 2023-10-27 14:08:30,597 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-15) Removing 0 from errorTimes 2023-10-27 14:08:30,597 WARN [ome.services.db.SelfCorrectingDataSource] (.Server-15) Registering error with list: Current size: 172 2023-10-27 14:08:30,597 ERROR [ome.services.db.SelfCorrectingDataSource] (.Server-15) Failed to acquire connection after retries=3 java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource 783e9342-628e-4aef-be4b-b32a8d81708e with 25 connection(s) (0 still available) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:332) at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:86) at ome.services.db.SelfCorrectingDataSource.call(SelfCorrectingDataSource.java:124) at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:97) at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy100.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) at ome.services.blitz.impl.ServiceFactoryI.getQueryService(ServiceFactoryI.java:309) at omero.api._ServiceFactoryTie.getQueryService(_ServiceFactoryTie.java:185) at omero.api._ServiceFactoryDisp.___getQueryService(_ServiceFactoryDisp.java:626) at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1303) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource 783e9342-628e-4aef-be4b-b32a8d81708e still empty after 30s wait time at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:319) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328) ... 34 common frames omitted 2023-10-27 14:08:30,597 INFO [ org.perf4j.TimingLogger] (.Server-15) start[1698415600595] time[110002] tag[omero.call.exception] 2023-10-27 14:08:30,598 WARN [ ome.services.util.ServiceHandler] (.Server-15) CannotCreateTransactionException thrown. org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy100.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) at ome.services.blitz.impl.ServiceFactoryI.getQueryService(ServiceFactoryI.java:309) at omero.api._ServiceFactoryTie.getQueryService(_ServiceFactoryTie.java:185) at omero.api._ServiceFactoryDisp.___getQueryService(_ServiceFactoryDisp.java:626) at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1303) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: ome.conditions.DatabaseBusyException: Cannot acquire connection at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:113) at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246) ... 29 common frames omitted 2023-10-27 14:08:30,598 INFO [ ome.services.util.ServiceHandler] (.Server-15) Excp: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection 2023-10-27 14:08:30,598 ERROR [ ome.services.util.ServiceHandler] (.Server-15) Method interface ome.services.util.Executor$Work.doWork invocation took 110002 2023-10-27 14:08:30,602 INFO [o.services.sessions.SessionContext$Count] (.Server-28) -Reference count: 2c77f226-c9d0-4fd0-be6a-c17b2be114cf=12 2023-10-27 14:08:30,602 INFO [ omero.cmd.SessionI] (.Server-28) cleanupSelf(ServiceFactoryI(session-d9fcd50f-278a-4fc5-bb1c-d02164472eb0/2c77f226-c9d0-4fd0-be6a-c17b2be114cf)). 2023-10-27 14:08:31,400 DEBUG [ loci.formats.Memoizer] (.Server-25) start[1698415646948] time[64452] tag[loci.formats.Memoizer.setId] 2023-10-27 14:08:31,400 ERROR [ ome.io.bioformats.BfPixelBuffer] (.Server-25) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:31,400 ERROR [ ome.io.nio.PixelsService] (.Server-25) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1 at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79) at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124) at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:520) at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:650) at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:1498) at ome.services.ThumbnailCtx$2.actOnOneGroup(ThumbnailCtx.java:407) at ome.services.PerGroupActor.actOnByGroup(PerGroupActor.java:92) at ome.services.ThumbnailCtx.createAndPrepareMissingRenderingSettings(ThumbnailCtx.java:411) at ome.services.ThumbnailBean.getThumbnailByLongestSideSet(ThumbnailBean.java:1003) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216) at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1558.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.ThumbnailStoreI.getThumbnailByLongestSideSet_async(ThumbnailStoreI.java:83) at jdk.internal.reflect.GeneratedMethodAccessor1573.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy120.getThumbnailByLongestSideSet_async(Unknown Source) at omero.api._ThumbnailStoreTie.getThumbnailByLongestSideSet_async(_ThumbnailStoreTie.java:132) at omero.api._ThumbnailStoreDisp.___getThumbnailByLongestSideSet(_ThumbnailStoreDisp.java:743) at omero.api._ThumbnailStoreDisp.__dispatch(_ThumbnailStoreDisp.java:1088) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalArgumentException: Invalid index: -1 at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328) at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116) at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284) at loci.formats.FormatReader.setId(FormatReader.java:1443) at loci.formats.ImageReader.setId(ImageReader.java:849) at ome.io.nio.PixelsService$3.setId(PixelsService.java:869) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelFiller.setId(ChannelFiller.java:234) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.Memoizer.setId(Memoizer.java:690) at ome.io.bioformats.BfPixelsWrapper.(BfPixelsWrapper.java:52) at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73) ... 73 common frames omitted 2023-10-27 14:08:31,401 INFO [ ome.services.OmeroFilePathResolver] (.Server-25) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:31,406 INFO [ loci.formats.ImageReader] (.Server-25) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:31,483 DEBUG [ loci.formats.Memoizer] (l.Server-8) start[1698415643329] time[68153] tag[loci.formats.Memoizer.setId] 2023-10-27 14:08:31,483 ERROR [ ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:31,483 ERROR [ ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1 at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79) at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124) at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.services.ThumbnailBean$1.actOnOneGroup(ThumbnailBean.java:1032) at ome.services.PerGroupActor.actOnByGroup(PerGroupActor.java:92) at ome.services.ThumbnailBean.retrieveThumbnailSet(ThumbnailBean.java:1074) at ome.services.ThumbnailBean.getThumbnailByLongestSideSet(ThumbnailBean.java:1005) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216) at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1558.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.ThumbnailStoreI.getThumbnailByLongestSideSet_async(ThumbnailStoreI.java:83) at jdk.internal.reflect.GeneratedMethodAccessor1573.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy120.getThumbnailByLongestSideSet_async(Unknown Source) at omero.api._ThumbnailStoreTie.getThumbnailByLongestSideSet_async(_ThumbnailStoreTie.java:132) at omero.api._ThumbnailStoreDisp.___getThumbnailByLongestSideSet(_ThumbnailStoreDisp.java:743) at omero.api._ThumbnailStoreDisp.__dispatch(_ThumbnailStoreDisp.java:1088) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalArgumentException: Invalid index: -1 at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328) at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116) at loci.formats.in.ZarrReader.initFile(ZarrReader.java:284) at loci.formats.FormatReader.setId(FormatReader.java:1443) at loci.formats.ImageReader.setId(ImageReader.java:849) at ome.io.nio.PixelsService$3.setId(PixelsService.java:869) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelFiller.setId(ChannelFiller.java:234) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.Memoizer.setId(Memoizer.java:690) at ome.io.bioformats.BfPixelsWrapper.(BfPixelsWrapper.java:52) at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73) ... 70 common frames omitted 2023-10-27 14:08:31,484 WARN [ ome.services.ThumbnailBean] (l.Server-8) Retrieving thumbnail in set for Pixels ID 15837106 failed. ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.services.ThumbnailBean$1.actOnOneGroup(ThumbnailBean.java:1032) at ome.services.PerGroupActor.actOnByGroup(PerGroupActor.java:92) at ome.services.ThumbnailBean.retrieveThumbnailSet(ThumbnailBean.java:1074) at ome.services.ThumbnailBean.getThumbnailByLongestSideSet(ThumbnailBean.java:1005) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216) at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy119.getThumbnailByLongestSideSet(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1558.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.ThumbnailStoreI.getThumbnailByLongestSideSet_async(ThumbnailStoreI.java:83) at jdk.internal.reflect.GeneratedMethodAccessor1573.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy120.getThumbnailByLongestSideSet_async(Unknown Source) at omero.api._ThumbnailStoreTie.getThumbnailByLongestSideSet_async(_ThumbnailStoreTie.java:132) at omero.api._ThumbnailStoreDisp.___getThumbnailByLongestSideSet(_ThumbnailStoreDisp.java:743) at omero.api._ThumbnailStoreDisp.__dispatch(_ThumbnailStoreDisp.java:1088) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) 2023-10-27 14:08:31,485 INFO [ ome.services.OmeroFilePathResolver] (l.Server-8) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:31,490 INFO [ loci.formats.ImageReader] (l.Server-8) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml 2023-10-27 14:08:32,021 INFO [ome.services.db.SelfCorrectingDataSource] (2-thread-1) Found location in errorTimes: -6 2023-10-27 14:08:32,021 INFO [ome.services.db.SelfCorrectingDataSource] (2-thread-1) Removing 5 from errorTimes 2023-10-27 14:08:32,021 WARN [ome.services.db.SelfCorrectingDataSource] (2-thread-1) Registering error with list: Current size: 168 2023-10-27 14:08:32,021 INFO [ome.services.db.SelfCorrectingDataSource] (2-thread-1) Sleeping for 10000 then retry: 1 2023-10-27 14:08:32,819 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-38) Found location in errorTimes: -7 2023-10-27 14:08:32,819 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-38) Removing 6 from errorTimes 2023-10-27 14:08:32,819 WARN [ome.services.db.SelfCorrectingDataSource] (.Server-38) Registering error with list: Current size: 163 2023-10-27 14:08:32,819 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-38) Sleeping for 10000 then retry: 2 2023-10-27 14:08:34,732 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-37) Found location in errorTimes: -1 2023-10-27 14:08:34,732 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-37) Removing 0 from errorTimes 2023-10-27 14:08:34,732 WARN [ome.services.db.SelfCorrectingDataSource] (.Server-37) Registering error with list: Current size: 164 2023-10-27 14:08:34,733 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-37) Sleeping for 10000 then retry: 1 2023-10-27 14:08:36,615 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-31) Found location in errorTimes: -2 2023-10-27 14:08:36,615 INFO [ome.services.db.SelfCorrectingDataSource] (.Server-31) Removing 1 from errorTimes 2023-10-27 14:08:36,615 WARN [ome.services.db.SelfCorrectingDataSource] (.Server-31) Registering error with list: Current size: 164 2023-10-27 14:08:36,615 ERROR [ome.services.db.SelfCorrectingDataSource] (.Server-31) Failed to acquire connection after retries=3 java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource 783e9342-628e-4aef-be4b-b32a8d81708e with 25 connection(s) (0 still available) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:332) at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:86) at ome.services.db.SelfCorrectingDataSource.call(SelfCorrectingDataSource.java:124) at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:97) at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy100.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) at ome.services.blitz.impl.ServiceFactoryI.getQueryService(ServiceFactoryI.java:309) at omero.api._ServiceFactoryTie.getQueryService(_ServiceFactoryTie.java:185) at omero.api._ServiceFactoryDisp.___getQueryService(_ServiceFactoryDisp.java:626) at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1303) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource 783e9342-628e-4aef-be4b-b32a8d81708e still empty after 30s wait time at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:319) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181) at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150) at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328) ... 34 common frames omitted 2023-10-27 14:08:36,616 INFO [ org.perf4j.TimingLogger] (.Server-31) start[1698415606613] time[110002] tag[omero.call.exception] 2023-10-27 14:08:36,616 WARN [ ome.services.util.ServiceHandler] (.Server-31) CannotCreateTransactionException thrown. org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy100.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) at ome.services.blitz.impl.ServiceFactoryI.getQueryService(ServiceFactoryI.java:309) at omero.api._ServiceFactoryTie.getQueryService(_ServiceFactoryTie.java:185) at omero.api._ServiceFactoryDisp.___getQueryService(_ServiceFactoryDisp.java:626) at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1303) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: ome.conditions.DatabaseBusyException: Cannot acquire connection ```

Going to restart server...

will-moore commented 8 months ago

I can't even view previously viewable idr0125 images now! ResourceError etc....

SQ00015152__2016-06-09T03_29_42-Measurement

Screenshot 2023-10-27 at 16 20 29

Screenshot 2023-10-27 at 16 22 09

Current ZarrReader.jar matches the one used to import those plates: OMEZarrReader-0.3.1-jar-with-dependencies.jar

will-moore commented 8 months ago

Start again! Import a few new plates with ZarrReader 0.3.1 and use these to test rendering settings workflow...

Need to edit bulk.yml to comment out # exclude: "clientpath" since we are importing from same location as before... Also reset plates.yml to full list of plates Rename Screen to idr0125-way-cellpainting/screenA - Oct 2022 so a new one will be created by import...

omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231030.log  --errs /tmp/idr0125_20231030.err
will-moore commented 8 months ago

Plate: 11051 imported... update symlinks...

python idr-utils/scripts/managed_repo_symlinks.py Plate:11051 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

fileset_dirs {}

Fileset: 5288471 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2023-10/30/11-46-57.796/
Render Image 16024912
fs_contents ['SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2023-10/30/11-46-57.796/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Apply rendering settings to 1 image.. 16021651

omero render set Image:16021651 screenA/idr0125-screenA-renderdef.yml

Copy from 1 image to plate 16:38...

omero --debug=2 render copy Image:16021651 Plate:11051 > /tmp/idr0125_PR59_copyplate_11051.log 2>/tmp/idr0125_PR59_copyplate_11051.err

EDIT: 22:40 (6 hrs):

$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
   1475    7385   57595

$ tail -f /tmp/idr0125_PR59_copyplate_11051.err
Image:16023646 got thumbnail in 13.73s
Image:16023647 got thumbnail in 14.26s
Image:16023648 got thumbnail in 12.48s
Image:16023631 got thumbnail in 13.94s
Image:16023632 got thumbnail in 14.10s
Image:16023633 got thumbnail in 13.19s
Image:16023634 got thumbnail in 13.87s
Image:16023635 got thumbnail in 12.93s
Image:16023636 got thumbnail in 13.69s
Image:16023637 got thumbnail in 14.83s

Batches of thumbnails being updated periodically... Last one at 22:24.

$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 57K Oct 30 22:24 /tmp/idr0125_PR59_copyplate_11051.err
$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 65K Oct 30 23:12 /tmp/idr0125_PR59_copyplate_11051.err
$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
   1686    8440   65824
$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 73K Oct 31 00:00 /tmp/idr0125_PR59_copyplate_11051.err
$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
   1897    9495   74053

25 mins later, no change! Batches updated quite infrequently!

...Done after 13 hours 20 mins

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
   3458   17295  134893
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 132K Oct 31 06:02 /tmp/idr0125_PR59_copyplate_11051.err
will-moore commented 8 months ago

Since there was long waits between setting batch generation of thumbnails (default batch=100), let's simply do 1 thumbnail at a time. There doesn't seem to be any advantage of doing this in batches, since for each batch, the thumbnails are generated 1 at a time.

Update symlinks on a bunch of newly-imported plates (before viewing them)... (import still running)

9:50...

for i in {11052..11058}; do
    echo "$i"
    python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
...

Fileset: 5288478 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-16/2023-10/31/05-52-01.142/
Render Image 16047490
fs_contents ['SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-16/2023-10/31/05-52-01.142/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr

...slow but all completed OK

Picked first image from Plate: 11052... Updated to https://github.com/ome/omero-cli-render/pull/59/commits/b8fba620a2b882050c1858f3a7d8bee8c8efdede 10:49...

omero --debug=2 render set Image:16025107 screenA/idr0125-screenA-renderdef.yml 
$ omero --debug=2 render copy Image:16025107 Plate:11052 > /tmp/idr0125_PR59_copyplate_11052.log 2>/tmp/idr0125_PR59_copyplate_11052.err

...11:05 - still nothing in /tmp/idr0125_PR59_copyplate_11052.err!

At 12:21... - no updates to the err since 11:39!

(base) bash-4.2$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
    209    1055    8221
(base) bash-4.2$ 
(base) bash-4.2$ tail -f /tmp/idr0125_PR59_copyplate_11052.err
Image:16028356 got thumbnail in 15.48s
Image:16028357 got thumbnail in 13.78s
Image:16028358 got thumbnail in 12.15s
Image:16028359 got thumbnail in 13.84s
Image:16028360 got thumbnail in 11.72s
Image:16028361 got thumbnail in 14.74s
Image:16028362 got thumbnail in 15.51s
Image:16028363 got thumbnail in 13.39s
Image:16028364 got thumbnail in 12.54s
Image:16028347 got thumbnail in 18.88s
(base) bash-4.2$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 8.1K Oct 31 11:39 /tmp/idr0125_PR59_copyplate_11052.err

At 15:50... - no .err update since 15:01.

$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
   1053    5275   41137
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 41K Oct 31 15:01 /tmp/idr0125_PR59_copyplate_11052.err

23:00...

$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
   2952   14770  115197
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 113K Oct 31 22:35 /tmp/idr0125_PR59_copyplate_11052.err

Done in 13 hours 40 mins - slightly slower than before! 

$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
   3458   17295  134893
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  1 00:32 /tmp/idr0125_PR59_copyplate_11052.err

Image:16025108 got thumbnail in 14.12s
Image:16025109 got thumbnail in 13.35s
Image:16025110 got thumbnail in 13.87s
Image:16025111 got thumbnail in 14.31s
Image:16025112 got thumbnail in 12.92s
Image:16025113 got thumbnail in 14.49s
Image:16025114 got thumbnail in 24.59s
Image:16025115 got thumbnail in 14.69s
Image:16025116 got thumbnail in 14.56s

13 hours (3456 images) is 13.5 secs per image - which fits with the logs. Just strange that the logs aren't updated very frequently.

will-moore commented 8 months ago

13 hours to get thumbnails for a plate is better than previous rates of >2days per plate: https://idr-redmine.openmicroscopy.org/issues/228?issue_count=1&issue_position=1#note-109

Need to know if we can copy settings from an Image in one plate to a different Plate (didn't work before with messed-up channels). Also try applying settings in multiple treads as before (see redmine link) in multiple terminal screens.

First, update symlinks for newly imported plates... (import still running)...

for i in {11059..11084}; do
    echo "$i"
    python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
...

After an 1hr 20mins, done 24/27 links... - At this rate, it will take 7.5 hours to do all 136 Plates!

...
Fileset: 5288502 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/00-52-58.990/
Render Image 16129507
fs_contents ['SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/00-52-58.990/SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr
11083
fileset_dirs {}

Fileset: 5288503 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/03-34-08.818/
Render Image 16134402
...
will-moore commented 8 months ago

..Also want to compare whether omero render copy Image:ID Plate:ID (uses settingsService to set whole plate at once) is faster than `omero render set Plate:ID rendering.yml (saves settings to one image at at time).

Let's use a batch of 5 plates for each...

Pick an image from first Plate:11059, apply settings to that Image, then use it to apply settings to 5 Plates (including it's own plate)...

omero --debug=2 render set Image: 16049299 screenA/idr0125-screenA-renderdef.yml 

for i in {11059..11063}; do
    echo "$i"
    omero --debug=2 render copy Image:16049299 Plate:$i > /tmp/idr0125_PR59_copyplate_$i.log 2>/tmp/idr0125_PR59_copyplate_$i.err
done

In another screen..

for i in {11064..11068}; do
    echo "$i"
    omero --debug=2 render set Plate:$i screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_setplate_$i.log 2>/tmp/idr0125_PR59_setplate_$i.err
done

EDIT: few days later... For copy plate...

$ ls -alh /tmp/idr0125_PR59_cop*
-rw-r--r--. 1 omero-server omero-server 132K Oct 31 06:02 /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  1 00:32 /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  4 01:19 /tmp/idr0125_PR59_copyplate_11059.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  4 14:16 /tmp/idr0125_PR59_copyplate_11060.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  5 02:57 /tmp/idr0125_PR59_copyplate_11061.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  5 15:44 /tmp/idr0125_PR59_copyplate_11062.err
-rw-r--r--. 1 omero-server omero-server 132K Nov  6 04:17 /tmp/idr0125_PR59_copyplate_11063.err

Thumbnails look good for plate:11063. From logs it looks like thumbnails took 13 hours per Plate, and copying from an Image on one plate to a different Plate worked OK! 👍

For set...

$ ls -alh /tmp/idr0125_PR59_set* | grep err
-rw-r--r--. 1 omero-server omero-server 284K Nov  4 11:12 /tmp/idr0125_PR59_setplate_11064.err
-rw-r--r--. 1 omero-server omero-server 284K Nov  5 09:39 /tmp/idr0125_PR59_setplate_11065.err
-rw-r--r--. 1 omero-server omero-server 284K Nov  6 08:00 /tmp/idr0125_PR59_setplate_11066.err
-rw-r--r--. 1 omero-server omero-server 281K Nov  7 06:51 /tmp/idr0125_PR59_setplate_11067.err
-rw-r--r--. 1 omero-server omero-server 1.1K Nov  7 06:51 /tmp/idr0125_PR59_setplate_11068.err

This looks like it took about 22 hours for each plate and it was only midway through the penultimate plate when I restarted the server below on the 7th Nov.

Will restart thumbnailing using copy command below...

will-moore commented 8 months ago

It looks like the most recent OMEZarrReader fixes had addressed the channel issues seen above. Stopped import (after 67 Plates) - last plate imported was SQ00015138__2016-06-11T17_59_08-Measurement1.

Updated ZarrReader to today's merge build https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/673/console Deleted 67 lines from idr0125-plates.tsv and restarted import...

omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231107.log  --errs /tmp/idr0125_20231107.err
will-moore commented 8 months ago

Symlinks for newly imported plates... 10:15...

for i in {11085..11117}; do
    echo "$i"
    python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done

EDIT: failed with DatabaseBusyException since I am importing and viewing in webcilent at the same time. Re-ran...

After Plate:11117 (last plate imported before server restart), IDs jump to Plate:11151...

Plates that still need thumbnailing...

11053-11058 11067-11117 11151-...