Open will-moore opened 1 year ago
Updated bulk import yaml to import METADATA.ome.xml
for every plate...
I went run bulk import on idr0125-pilot
...
omero import --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20230918.log --errs /tmp/idr0125_20230918.err
but found that this failed because every Plate.zarr directory is empty.
ls /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/
Could this be due to the update of symlinks in ManagedRepo? Commands are at https://github.com/IDR/idr0125-way-cellpainting/blob/d883ec3a88178b3aaf207acbf9612ef0e5e3f840/scripts/symlinks.bash
E.g.
cd /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-83/2022-12/14/10-51-00.924
rm -rf SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
ln -s /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
But this shouldn't affect what's under /uod/idr/filesets/idr0125-way-cellpainting/s3-20221212/
??
Instead, lets simply download and import the OME/METADATA.ome.xml
files...
NB: this loop repeats for every -
in each yaml row as well as the actual plate names!
for r in $(cat /uod/idr/metadata/idr0125-way-cellpainting/docs/_data/plates.yml); do
cd
cd idr0125
echo $r
mkdir -p $r
cd $r
wget https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$r/OME/METADATA.ome.xml
done
Needed to rename each dir to remove .ome.zarr
because trying to import e.g. SQ00015117__2016-04-13T18_12_45-Measurement1.ome.zarr/METADATA.ome.xml
failed looking for .zgroup
etc.
for i in $(ls ./); do
mv $i ${i/.ome.zarr/}
done
for i in $(ls ./); do
omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log --errs /tmp/$i.err
done
$ tail -f /tmp/SQ00014812__2016-05-23T20_44_31-Measurement1.err
2023-09-18 19:01:25,306 3563454 [l.Client-3] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 52313944
2023-09-18 19:01:26,455 3564603 [l.Client-2] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1/METADATA.ome.xml
Other imported objects:
Fileset:5287672
==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 0:59:19.730
Seems like import got stopped at some point (maybe due to server restart)... Last log files are:
-rw-r--r--. 1 omero-server omero-server 4.3K Sep 19 12:33 SQ00015050__2016-05-24T06_38_27-Measurement1.err
-rw-r--r--. 1 omero-server omero-server 0 Sep 19 11:54 SQ00015050__2016-05-24T06_38_27-Measurement1.log
Moved the imported dirs out of idr0125/
into idr00125_imported/
ls idr0125_imported/
SQ00014812__2016-05-23T20_44_31-Measurement1 SQ00014816__2016-05-23T14_07_55-Measurement1 SQ00014820__2016-05-25T21_16_25-Measurement1 SQ00015044__2016-05-20T05_34_55-Measurement1 SQ00015048__2016-05-25T22_53_31-Measurement1
SQ00014813__2016-05-23T19_03_28-Measurement1 SQ00014817__2016-05-23T12_08_18-Measurement1 SQ00015041__2016-05-25T19_39_25-Measurement1 SQ00015045__2016-05-20T03_57_42-Measurement1 SQ00015049__2016-05-24T08_17_26-Measurement1
SQ00014814__2016-05-23T17_24_56-Measurement1 SQ00014818__2016-05-23T10_30_25-Measurement1 SQ00015042__2016-05-25T18_02_08-Measurement1 SQ00015046__2016-05-20T02_20_03-Measurement1 SQ00015050__2016-05-24T06_38_27-Measurement1
SQ00014815__2016-05-23T15_46_08-Measurement1 SQ00014819__2016-05-23T08_51_51-Measurement1 SQ00015043__2016-05-25T16_25_21-Measurement1 SQ00015047__2016-05-20T00_41_45-Measurement1
and restarted import in for loop as above, from idr0125/
Moved SQ00015110__2016-05-05T04_50_06-Measurement1
and other Plates before it into idr0125_imported
and restarted import from idr0125
as above...
for i in $(ls ./); do omero import --transfer=ln_s --skip=all --name=$i $i/METADATA.ome.xml --file /tmp/$i.log --errs /tmp/$i.err
For running mkngff
on idr0125-pilot, we have data mounted at
$ ls /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/
Lets run mkngff on this Plate/Filesets... as regular wmoore
user on idr0125-pilot...
(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ omero mkngff sql --secret=$SECRET 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr > idr0125_sql/5287672.sql
Using session for public@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024 for fileset: 5287672
...
$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287672.sql
UPDATE 3456
BEGIN
mkngff_fileset
----------------
5288197
(1 row)
COMMIT
as omero-server...
(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287672 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-4/2023-09/18/18-02-07.024_mkngff/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Viewing... http://localhost:1040/webclient/?show=well-2339987
EDIT: after 3 days this image(s) are still not viewable. Probably just takes too long for ZarrReader to read a BIG plate with all the data being remote in USA.
All METADATA.ome.xml imported above.
Last batch - first plate SQ00015111__2016-05-05T03_10_03-Measurement1
created at 2023-09-28 15:02:03
Last plate imported (created) at 2023-10-02 10:23:45
.
94 Plates imported in about 90 hours. 1 hour per plate!
Since running mkngff above, and attempting to view image at http://localhost:1040/webclient/?show=well-2339987 (which still isn't visible) we now have ZarrReader update on idr0125-pilot.
Try another plate... 2nd plate: SQ00014813__2016-05-23T19_03_28-Measurement1
Fileset ID: 5287673
As wmoore user on idr0125-pilot...
omero mkngff sql --secret=$SECRET 5287673 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr > idr0125_sql/5287673.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585 for fileset: 5287673
...
(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ psql -U omero -d idr -h $DBHOST -f idr0125_sql/5287673.sql
UPDATE 3456
BEGIN
mkngff_fileset
----------------
5288270
(1 row)
COMMIT
As omero-server user...
(venv3) (base) bash-4.2$ omero mkngff symlink /data/OMERO/ManagedRepository 5287673 "/cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr"
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-09/18/19-01-33.585_mkngff/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr
View image at http://localhost:1040/webclient/?show=image-14847759 ...
This image is also still not viewable. I think we have to conclude that this approach isn't going to work. There's probably just too many files to load remotely (from the US).
Since the size of a "metadata-only" plate appears to be moderate, we can look again at the workflow tested earlier:
(base) [wmoore@pilot-idr0125-omeroreadwrite idr0125]$ du -sh ./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/
82M ./SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/
With idr0125_plates.csv:
SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr
SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr
SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr
SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr
SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr
SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr
SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr
SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr
SQ00015041__2016-05-25T19_39_25-Measurement1.ome.zarr
SQ00015042__2016-05-25T18_02_08-Measurement1.ome.zarr
SQ00015043__2016-05-25T16_25_21-Measurement1.ome.zarr
SQ00015044__2016-05-20T05_34_55-Measurement1.ome.zarr
SQ00015045__2016-05-20T03_57_42-Measurement1.ome.zarr
SQ00015046__2016-05-20T02_20_03-Measurement1.ome.zarr
SQ00015047__2016-05-20T00_41_45-Measurement1.ome.zarr
SQ00015048__2016-05-25T22_53_31-Measurement1.ome.zarr
SQ00015049__2016-05-24T08_17_26-Measurement1.ome.zarr
SQ00015050__2016-05-24T06_38_27-Measurement1.ome.zarr
SQ00015051__2016-05-24T05_00_04-Measurement1.ome.zarr
SQ00015052__2016-05-24T03_21_49-Measurement1.ome.zarr
SQ00015053__2016-05-24T01_42_05-Measurement1.ome.zarr
SQ00015054__2016-05-24T00_02_16-Measurement1.ome.zarr
SQ00015055__2016-05-23T22_23_05-Measurement1.ome.zarr
SQ00015056__2016-05-20T16_27_19-Measurement1.ome.zarr
SQ00015057__2016-05-20T14_48_37-Measurement1.ome.zarr
SQ00015058__2016-05-20T08_50_58-Measurement1.ome.zarr
SQ00015059__2016-05-20T07_12_47-Measurement1.ome.zarr
SQ00015096__2016-06-08T17_05_23-Measurement1.ome.zarr
SQ00015097__2016-06-08T15_26_27-Measurement1.ome.zarr
SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr
SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr
SQ00015100__2016-05-16T22_01_19-Measurement1.ome.zarr
SQ00015101__2016-05-16T20_23_02-Measurement1.ome.zarr
SQ00015102__2016-05-16T18_45_00-Measurement1.ome.zarr
SQ00015103__2016-05-16T17_06_53-Measurement1.ome.zarr
SQ00015105__2016-05-06T00_10_33-Measurement1.ome.zarr
SQ00015106__2016-05-05T22_30_28-Measurement1.ome.zarr
SQ00015107__2016-05-05T09_50_51-Measurement1.ome.zarr
SQ00015108__2016-05-05T08_10_09-Measurement1.ome.zarr
SQ00015109__2016-05-05T06_29_59-Measurement1.ome.zarr
SQ00015110__2016-05-05T04_50_06-Measurement1.ome.zarr
SQ00015111__2016-05-05T03_10_03-Measurement1.ome.zarr
SQ00015112__2016-05-05T01_29_08-Measurement1.ome.zarr
SQ00015116__2016-04-13T16_31_49-Measurement1.ome.zarr
SQ00015117__2016-04-13T18_12_45-Measurement1.ome.zarr
SQ00015118__2016-04-13T19_52_28-Measurement1.ome.zarr
SQ00015119__2016-04-13T21_33_42-Measurement1.ome.zarr
SQ00015120__2016-04-13T23_14_32-Measurement1.ome.zarr
SQ00015121__2016-04-14T00_54_07-Measurement1.ome.zarr
SQ00015122__2016-04-14T02_33_42-Measurement1.ome.zarr
SQ00015123__2016-04-14T04_14_45-Measurement1.ome.zarr
SQ00015124__2016-04-14T07_35_09-Measurement1.ome.zarr
SQ00015125__2016-04-13T14_53_08-Measurement2.ome.zarr
SQ00015126__2016-03-25T14_15_31-Measurement1.ome.zarr
SQ00015127__2016-04-14T05_53_59-Measurement1.ome.zarr
SQ00015128__2016-04-14T21_08_47-Measurement1.ome.zarr
SQ00015129__2016-04-15T05_30_25-Measurement1.ome.zarr
SQ00015130__2016-04-01T13_10_04-Measurement1.ome.zarr
SQ00015131__2016-03-25T17_23_46-Measurement1.ome.zarr
SQ00015132__2016-04-24T07_20_38-Measurement1.ome.zarr
SQ00015133__2016-04-24T08_59_07-Measurement1.ome.zarr
SQ00015134__2016-06-12T00_30_52-Measurement1.ome.zarr
SQ00015135__2016-06-11T22_53_05-Measurement1.ome.zarr
SQ00015136__2016-06-11T21_14_31-Measurement1.ome.zarr
SQ00015137__2016-06-11T19_36_55-Measurement1.ome.zarr
SQ00015138__2016-06-11T17_59_08-Measurement1.ome.zarr
SQ00015139__2016-06-11T16_21_04-Measurement1.ome.zarr
SQ00015140__2016-06-11T14_43_11-Measurement1.ome.zarr
SQ00015141__2016-06-09T08_43_58-Measurement1.ome.zarr
SQ00015142__2016-05-19T23_03_33-Measurement1.ome.zarr
SQ00015143__2016-05-19T21_25_23-Measurement1.ome.zarr
SQ00015144__2016-05-19T19_47_38-Measurement1.ome.zarr
SQ00015145__2016-05-19T18_10_43-Measurement1.ome.zarr
SQ00015146__2016-05-19T10_37_52-Measurement1.ome.zarr
SQ00015147__2016-05-17T07_49_53-Measurement1.ome.zarr
SQ00015148__2016-05-17T01_17_02-Measurement1.ome.zarr
SQ00015149__2016-05-16T23_39_00-Measurement1.ome.zarr
SQ00015150__2016-06-09T06_45_35-Measurement1.ome.zarr
SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr
SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr
SQ00015153__2016-06-09T01_50_39-Measurement1.ome.zarr
SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr
SQ00015155__2016-04-14T17_48_56-Measurement1.ome.zarr
SQ00015156__2016-04-14T19_29_39-Measurement1.ome.zarr
SQ00015157__2016-04-14T22_48_58-Measurement1.ome.zarr
SQ00015158__2016-04-15T00_30_51-Measurement1.ome.zarr
SQ00015159__2016-04-15T02_10_36-Measurement1.ome.zarr
SQ00015160__2016-04-15T03_50_42-Measurement1.ome.zarr
SQ00015162__2016-04-15T08_49_32-Measurement1.ome.zarr
SQ00015163__2016-04-20T21_00_28-Measurement1.ome.zarr
SQ00015164__2016-04-20T22_38_54-Measurement1.ome.zarr
SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr
SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr
SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr
SQ00015168__2016-04-21T05_12_27-Measurement1.ome.zarr
SQ00015169__2016-04-21T06_50_28-Measurement1.ome.zarr
SQ00015170__2016-04-21T08_29_08-Measurement1.ome.zarr
SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr
SQ00015172__2016-04-21T18_01_12-Measurement1.ome.zarr
SQ00015173__2016-04-15T07_09_23-Measurement1.ome.zarr
SQ00015194__2016-03-25T12_39_04-Measurement1.ome.zarr
SQ00015195__2016-04-24T21_41_05-Measurement2.ome.zarr
SQ00015196__2016-04-24T23_20_24-Measurement2.ome.zarr
SQ00015197__2016-04-24T13_30_35-Measurement1.ome.zarr
SQ00015198__2016-04-24T15_10_23-Measurement1.ome.zarr
SQ00015199__2016-04-23T15_43_06-Measurement1.ome.zarr
SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr
SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr
SQ00015202__2016-04-23T22_17_34-Measurement1.ome.zarr
SQ00015203__2016-04-25T00_59_49-Measurement2.ome.zarr
SQ00015204__2016-04-24T20_02_01-Measurement2.ome.zarr
SQ00015205__2016-04-24T02_21_50-Measurement1.ome.zarr
SQ00015206__2016-04-24T04_00_27-Measurement1.ome.zarr
SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr
SQ00015208__2016-04-22T16_36_31-Measurement1.ome.zarr
SQ00015209__2016-04-22T18_16_23-Measurement1.ome.zarr
SQ00015210__2016-04-25T19_58_35-Measurement1.ome.zarr
SQ00015211__2016-04-23T17_22_06-Measurement1.ome.zarr
SQ00015212__2016-04-23T19_01_00-Measurement1.ome.zarr
SQ00015214__2016-05-04T23_48_48-Measurement1.ome.zarr
SQ00015215__2016-05-04T18_49_26-Measurement1.ome.zarr
SQ00015216__2016-05-04T17_08_56-Measurement1.ome.zarr
SQ00015217__2016-05-04T22_08_57-Measurement1.ome.zarr
SQ00015218__2016-05-04T08_36_42-Measurement1.ome.zarr
SQ00015219__2016-05-04T15_29_24-Measurement1.ome.zarr
SQ00015220__2016-05-04T20_29_20-Measurement1.ome.zarr
SQ00015221__2016-05-04T06_58_09-Measurement1.ome.zarr
SQ00015222__2016-05-06T05_09_24-Measurement1.ome.zarr
SQ00015223__2016-05-06T03_29_13-Measurement1.ome.zarr
SQ00015224__2016-05-06T01_49_03-Measurement1.ome.zarr
SQ00015229__2016-05-13T08_10_01-Measurement1.ome.zarr
SQ00015230__2016-05-12T18_00_11-Measurement1.ome.zarr
SQ00015231__2016-05-12T09_02_26-Measurement1.ome.zarr
SQ00015232__2016-05-12T07_24_32-Measurement1.ome.zarr
SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr
(base) [wmoore@pilot-idr0125-omeroreadwrite ngff]$ df -h /ngff
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 100G 87G 14G 87% /
cd /ngff
for zarr in $(cat idr0125_plates.csv);
do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done
...
11:21am...
...completed 18:54. 7 hrs 33 mins
to download all metadata-only plates.
Used 9 GB
of space, based on what's left:
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ df -h /ngff
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 100G 96G 4.8G 96% /
Started import the metadata-only plates downloaded to idr0125-pilot
at /ngff/idr0125
using bulk.yml from https://github.com/IDR/idr0125-way-cellpainting/commit/8973828b3959e2e35b38acdbeec32c886bd91957
screen -r 17598.idr0125_import
...
(venv3) (base) bash-4.2$ omero import --depth=100 --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231011.log --errs /tmp/idr0125_20231011.err
...
2023-10-11 11:49:19,024 197526 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-10-11 11:49:22,703 201205 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-10-11 11:49:23,653 202155 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-10-11 11:49:27,030 205532 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
2023-10-11 11:49:27,307 205809 [ main] ERROR loci.formats.FormatHandler - ZarrReader returning list of used files of size: 11157
2023-10-11 11:49:28,816 207318 [ main] INFO ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 1 group(s) with 1 call(s) to setId in 171492ms. (197094ms total) [0 unknowns]
2023-10-11 11:49:29,672 208174 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-11 11:49:31,602 210104 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-11 11:49:33,942 212444 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-11 11:49:33,958 212460 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-11 11:49:33,958 212460 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-10-11 11:49:33,958 212460 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-10-11 11:49:33,959 212461 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-11 11:49:33,959 212461 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-11 11:49:33,959 212461 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-10-11 11:49:34,669 213171 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Screen:name:idr0125-way-cellpainting/screenA
2023-10-11 11:49:34,887 213389 [2-thread-1] INFO ome.formats.importer.ImportLibrary - Import target specifies container: Screen:3351
...
51 plates imported so far: ~ 10 per day.
Test import of first Plate on idr-testing with ZarrReader: /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_p643_b565.jar
from https://github.com/IDR/idr-metadata/issues/675#issuecomment-1755198551
Using aws
as wmoore in aws conda env...
$ aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Moved data to /ngff/idr0125/
and chown to omero-server.
Import as omero-server...
$ omero import --depth=100 --transfer=ln_s --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125.log --errs /tmp/idr0125.err
2023-10-17 10:57:54,260 291 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2023-10-17 10:57:54,280 311 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2023-10-17 10:57:54,344 375 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-10-17 10:57:54,345 376 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-10-17 10:57:54,345 376 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-10-17 10:57:54,345 376 [ main] INFO formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-10-17 10:57:54,345 376 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-10-17 10:57:54,348 379 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-17 10:57:54,716 747 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-10-17 10:58:03,322 9353 [ main] ERROR loci.formats.FormatHandler - ZarrReader getUsed files, skipPixels: true
2023-10-17 10:58:03,322 9353 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
2023-10-17 10:58:03,329 9360 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/0/.zarray
2023-10-17 10:58:03,330 9361 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zgroup
2023-10-17 10:58:03,330 9361 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zattrs
...
2023-10-17 10:58:03,653 9684 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zattrs
2023-10-17 10:58:03,653 9684 [ main] ERROR loci.formats.FormatHandler - Adding to the used files list: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/.zgroup
2023-10-17 10:58:04,482 10513 [ main] INFO ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 0 group(s) with 1 call(s) to setId in 8723ms. (9766ms total) [0 unknowns]
2023-10-17 10:58:04,530 10561 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-17 10:58:05,062 11093 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-17 10:58:05,450 11481 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-17 10:58:05,464 11495 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.99.1.el7.x86_64
No imports found
Also on idr0125-pilot, wanted to test whether import with chunks is working with current ZarrReader (which is likely 0.3.1 and Canonical paths PR)... EDIT - probably this is Performace PR before getUsedFiles - this works for import and for viewing mkngff Filesets after memo is generated - but doesn't work for generating memo file after mkngff.
Import idr0054 data directly from goofys mount of BIA bucket...
(venv3) (base) bash-4.2$ omero import -d 15301 --depth=100 /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054.log --errs /tmp/idr0054.err
Or from local copy...
omero import -d 15301 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054a.log --errs /tmp/idr0054a.err
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/OMEZarrReader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/omero/server/OMERO.server-5.6.7-ice36/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2023-10-17 12:39:08,698 300 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-10-17 12:39:08,717 319 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 0.3.2-SNAPSHOT revision: e4b0b24f6f1cc9b98ea7edc03f218a631d79d0d2 date: 20230925-0010
2023-10-17 12:39:08,782 384 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-10-17 12:39:09,251 853 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-10-17 12:39:09,275 877 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to initialize file: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-17 12:39:09,280 882 [ main] ERROR loci.formats.FormatHandler - ZarrReader initializing ZarrService: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,286 888 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing existing OME-XML
2023-10-17 12:39:09,679 1281 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing top level group attributes: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,694 1296 [ main] ERROR loci.formats.FormatHandler - ZarrReader generating group keys
2023-10-17 12:39:09,694 1296 [ main] ERROR loci.formats.FormatHandler - ZarrReader adding group keys from ZarrService
2023-10-17 12:39:09,729 1331 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing group Keys
2023-10-17 12:39:09,736 1338 [ main] ERROR loci.formats.FormatHandler - ZarrReader attempting to generate Array Keys
2023-10-17 12:39:09,736 1338 [ main] ERROR loci.formats.FormatHandler - ZarrReader adding Array Keys from ZarrService
2023-10-17 12:39:09,749 1351 [ main] ERROR loci.formats.FormatHandler - ZarrReader ordering Array Paths
2023-10-17 12:39:09,749 1351 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/0
2023-10-17 12:39:09,749 1351 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,764 1366 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,765 1367 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/1
2023-10-17 12:39:09,765 1367 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 1 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/1
2023-10-17 12:39:09,766 1368 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,766 1368 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/2
2023-10-17 12:39:09,766 1368 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,767 1369 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 2 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/2
2023-10-17 12:39:09,767 1369 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,767 1369 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/3
2023-10-17 12:39:09,768 1370 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,768 1370 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 3 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/3
2023-10-17 12:39:09,769 1371 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,769 1371 [ main] ERROR loci.formats.FormatHandler - ZarrReader setting metadata for series : 0/4
2023-10-17 12:39:09,769 1371 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 0 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/0
2023-10-17 12:39:09,770 1372 [ main] ERROR loci.formats.FormatHandler - Opening zarr for series 4 at path: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4
2023-10-17 12:39:09,770 1372 [ main] ERROR loci.formats.FormatHandler - ZarrReader opening Zarr to get Shape
2023-10-17 12:39:09,771 1373 [ main] ERROR loci.formats.FormatHandler - ZarrReader populating Pixels metadata
2023-10-17 12:39:09,771 1373 [ main] ERROR loci.formats.FormatHandler - ZarrReader parsing plate
2023-10-17 12:39:09,771 1373 [ main] ERROR loci.formats.FormatHandler - ZarrReader initialization complete
2023-10-17 12:39:09,772 1374 [ main] ERROR loci.formats.FormatHandler - ZarrReader fetching list of used files: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
2023-10-17 12:39:09,790 1392 [ main] ERROR loci.formats.FormatHandler - ZarrReader returning list of used files of size: 444
2023-10-17 12:39:09,835 1437 [ main] INFO ome.formats.importer.ImportCandidates - 444 file(s) parsed into 1 group(s) with 1 call(s) to setId in 540ms. (584ms total) [0 unknowns]
2023-10-17 12:39:09,884 1486 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-10-17 12:39:10,415 2017 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-10-17 12:39:10,744 2346 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-10-17 12:39:10,756 2358 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-10-17 12:39:10,756 2358 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-10-17 12:39:10,756 2358 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-10-17 12:39:10,756 2358 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-10-17 12:39:10,757 2359 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-10-17 12:39:10,757 2359 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-10-17 12:39:11,315 2917 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Dataset:15301
2023-10-17 12:39:11,660 3262 [2-thread-1] INFO ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:15301
2023-10-17 12:39:11,670 3272 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-10-17 12:39:34,326 25928 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-10-17 12:39:34,459 26061 [3-thread-1] INFO ts.importer.transfers.UploadFileTransfer - Transferring /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml...
2023-10-17 12:39:34,503 26105 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-10-17 12:39:34,735 26337 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
...
2023-10-17 12:42:18,693 190295 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/0/4/0/9/0/0/0
2023-10-17 12:42:22,352 193954 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-10-17 12:42:22,477 194079 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 53840675
2023-10-17 12:42:24,570 196172 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 53840675
2023-10-17 12:42:34,559 206161 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 53840675
2023-10-17 12:42:36,168 207770 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 53840675
2023-10-17 12:42:36,252 207854 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 53840675
2023-10-17 12:42:36,463 208065 [l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 53840675
2023-10-17 12:42:37,860 209462 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0054/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5288333
==> Summary
444 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:03:26.856
Since import from https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1757512153 got interrupted (server restarts above), 61 plates were imported, so deleted 61 rows from idr0125-screenA-bulk.yml
- then an extra row since that was SQ00015133__2016-04-24T08_59_07-Measurement1
which was the last plate imported.
Then restarted import...
(venv3) (base) bash-4.2$ omero import --depth=100 --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231018.log --errs /tmp/idr0125_20231018.err
EDIT: oops - forgot to use --transfer=ln_s
here!
Importing metadata-only plates is still in progress on idr0125-pilot.
Testing symlinking etc... with https://github.com/IDR/idr-utils/pull/54
Moved some imported Plates into temp Screen:3352
to use for symlinking...
As omero-server user...
$ cd /uod/idr/metadata
$ python idr-utils/scripts/managed_repo_symlinks.py Screen:3352 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
fileset_dirs {}
Fileset: 5288271 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-29/2023-10/11/11-57-46.608/
Render Image 15325028
fs_contents ['ngff']
Symlink target not found: /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/ngff
Fileset: 5288272 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/
Render Image 15331556
^C
A couple of issues:
ngff/idr0125/
after the prefix and before Plate name, e.g. demo_2/Blitz-0-Ice.ThreadPool.Server-29/2023-10/11/11-57-46.608/ngff/idr0125/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/
. Those extra paths aren't expected - we instead need to look inside them for the Plate which is to be replaced by a symlink.The 2nd issue (bullet point) above seems to be due to the path in plates.tsv
, e.g. /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
https://github.com/IDR/idr0125-way-cellpainting/blob/8973828b3959e2e35b38acdbeec32c886bd91957/screenA/idr0125-screenA-plates.tsv
However, the usual behaviour elsewhere is for the path/to/plate.zarr truncated to just give the plate.zarr, which is added to the Fileset prefix. managed_repo/fileset_prefix/plate.zarr
. This is what was happening for the first import...
This is probably because I didn't use --transfer=ln_s
for this import...
Stopped import (restarted the server and updated ZarrReader to use OMEZarrReader-0.3.1-jar-with-dependencies.jar
).
77 plates imported now. Last one was SQ00015150__2016-06-09T06_45_35-Measurement1
.
Updated sudo vi idr0125-screenA-plates.tsv
to remove 17 lines upto and including that plate.
Restart...
omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231020.log --errs /tmp/idr0125_20231020.err
For the first 77 plates imported, we need to use a TEMP commit of https://github.com/IDR/idr-utils/pull/54/commits/2888b21c2751421ecf24a845c0b0da2aa6662184 to add in the /ngff/idr0125/
to the filesetPrefix...
With this installed....
python idr-utils/scripts/managed_repo_symlinks.py Screen:3352 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
...
Fileset: 5288272 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/ngff/idr0125
Render Image 15331556
fs_contents ['SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2023-10/11/14-22-23.181/ngff/idr0125/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr
Fileset: 5288273 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2023-10/11/16-38-30.825/ngff/idr0125
Render Image 15338049
fs_contents ['SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2023-10/11/16-38-30.825/ngff/idr0125/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr
Something strange going on.....
ALL 3 of the images rendered above (one for the each fileset) are orphans! E.g http://localhost:1040/iviewer/image_data/15325028/ They aren't found by /webclient/?show=image-15325028 because http://localhost:1040/webclient/api/paths_to_object/?image=15325028 returns "orphaned".
"meta": {
"imageName": "SQ00014813__2016-05-23T19_03_28-Measurement1 [P/24/0/0]",
"imageDescription": "",
"imageAuthor": "Public data",
"projectName": "Multiple",
"projectId": null,
"projectDescription": "",
"datasetName": "Multiple",
"datasetId": null,
"datasetDescription": "",
"wellSampleId": "",
"wellId": "",
"imageTimestamp": 1464050222,
"imageId": 15325028,
"pixelsType": "uint16"
},
Images are found in that script with:
params.addId(fileset_id)
params.page(0, 1)
query = "select img from Image img where img.fileset.id=:id"
first_image = conn.getQueryService().findAllByQuery(query, params, conn.SERVICE_OPTS)[0]
It also appears that all plates imported above (without --transfer=ln_x
give ResourceError when trying to view an Image from any field position except the first (index=0). This is true both before and after the symlink update.
Seems that with the first 2 batches imported above (without in-place import) each Fileset creates lots of extra images: 6528
instead of 3456
.
6528 / 384 is 17 images per Well, which would be 2 for every field except the first. NB: Every field except the first gives ResourceError when trying to view.
(venv3) (base) bash-4.2$ grep -A 1 "Summary" /tmp/idr0125_20231011.err
...
==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 6528 images imported, 0 errors in 2:10:24.594
--
==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 6528 images imported, 0 errors in 2:12:56.714
--
==> Summary
11157 files uploaded, 0 filesets created, 0 images imported, 0 errors in 1:19:27.922
Will need to re-import all those 77 plates!
Checking last import logs /tmp/idr0125_20231020.err
found a couple of Plates failed to import:
/ngff/idr0125/...
Edited idr0125-screenA-plates.tsv
to try again with those 3 Plates, then with all others after SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr
screen -r idr0125_import
and cancelled import.
Restarted import... (still using OMEZarrReader-0.3.1-jar-with-dependencies.jar
)...
(venv3) (base) bash-4.2$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231023.log --errs /tmp/idr0125_20231023.err
EDIT... 3 failed imports re-tried above ALL failed with ClientPath match
- I guess the partial import still leaves Filesets.
$ python idr-utils/scripts/managed_repo_symlinks.py Screen:3351 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
fileset_dirs {}
Fileset: 5288371 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2023-10/20/13-55-33.718/
Render Image 15829653
fs_contents ['SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2023-10/20/13-55-33.718/SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015152__2016-06-09T03_29_42-Measurement1.ome.zarr
Fileset: 5288372 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-10/20/16-23-21.825/
Render Image 15831760
...
EDIT - Kinda slow to render Images and create symlinks: Took 75 mins, to do 17 Plates
Render all images in 17 plates...
cd /uod/idr/metadata/idr0125-way-cellpainting/
screen -r idr0125_symlinks
source /opt/omero/server/venv3/bin/activate
omero login
for a in {10951..10967};
do
echo "$a"
omero --debug=2 render set Plate:$a screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_$i.log 2>/tmp/idr0125_renderplate_$i.err;
done
EDIT - after 75 mins...
$ cat /tmp/idr0125_renderplate_10967.err | wc
573 2856 24626
$ tail /tmp/idr0125_renderplate_10967.err
...
Image:15830886 got thumbnail in 14.07s
Updated rendering settings for Image:15830869
Image:15830869 got thumbnail in 11.47s
Updated rendering settings for Image:15830870
Image:15830870 got thumbnail in 13.40s
Updated rendering settings for Image:15830871
Image:15830871 got thumbnail in 13.81s
Updated rendering settings for Image:15830872
Image:15830872 got thumbnail in 14.03s
Updated rendering settings for Image:15830873
Image:15830873 got thumbnail in 13.94s
EDIT - after nearly 5 hours
(base) [wmoore@pilot-idr0125-omeroreadwrite idr0125-way-cellpainting]$ cat /tmp/idr0125_renderplate_10967.err | wc
2117 10576 90246
Approx 1000 thumbnails rendered (from plate of 3456 images). This is VERY slow, compared to https://idr-redmine.openmicroscopy.org/issues/228?issue_count=1&issue_position=1#note-110 (9 months ago) e.g.
Image:14609144 got thumbnail in 0.68s
Updated rendering settings for Image:14609127
Previously thumbnail generation was 20 x faster than now!! Whereas when we "SaveAll" in webclient, all the 384 thumbnails (for 1 Well index) are generated in 10-15 mins.
EDIT - after 20 hours - not getting any faster!
$ cat /tmp/idr0125_renderplate_10967.err | wc
2310 11541 98459
$ tail -f /tmp/idr0125_renderplate_10967.err
...
Updated rendering settings for Image:15833469
Image:15833469 got thumbnail in 14.02s
Updated rendering settings for Image:15833470
Image:15833470 got thumbnail in 13.40s
Updated rendering settings for Image:15833453
Decided to cancel import, since it may be slowing down the Thumbnail generation and I want to try different ZarrReader...
Last Plate to be imported was SQ00015198__2016-04-24T15_10_23-Measurement1
10974
.
Updated ZarrReader to use OMEZarrReader_performance_allfiles.jar
from https://github.com/ome/ZarrReader/pull/64#issuecomment-1768612367 which works for NGFF Import.
Restarting server will also kill the thumbnail generation above.
First, test Thumbnail creation on a new plate...
for i in {10953..10967}; do
echo "$i"
omero --debug=2 render set Plate:$i screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_$i.log 2>/tmp/idr0125_renderplate_$i.err;
done
This gives...
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2023-10/21/00-27-26.892/SQ00015156__2016-04-14T19_29_39-Measurement1.ome.zarr/OME/METADATA.ome.xml
I guess that Plate that was imported with older ZarrReader - then image rendered and symlinks updated before trying to view images with latest ZarrReader.jar.
The OMEZarrReader_performance_allfiles.jar
doesn't work for viewing Plates imported with OMEZarrReader-0.3.1-jar-with-dependencies.jar
but let's at least test whether it works for Import...
Removed 10 rows from idr0125-screenA-plates.tsv
, including the last plate SQ00015198__2016-04-24T15_10_23-Measurement1
above, and restarted import...
$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231024.log --errs /tmp/idr0125_20231024.err
First plate imported with OMEZarrReader_performance_allfiles.jar
: SQ00015200__2016-04-01T08_46_01-Measurement1
(11001), render an Image in webclient (blank), then update symlinks for these 2 new plates:
Strangely, these plates have /ngff/idr0125/
in the Fileset path, even though they were imported with in-place import!
Used managed_repo_symlinks.py
TEMP commit to handle this as above...
$ python idr-utils/scripts/managed_repo_symlinks.py Plate:11001 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
Fileset: 5288421 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/24/14-43-25.039/ngff/idr0125
Render Image 15916984
fs_contents ['SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-10/24/14-43-25.039/ngff/idr0125/SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015200__2016-04-01T08_46_01-Measurement1.ome.zarr
$ python idr-utils/scripts/managed_repo_symlinks.py Plate:11002 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
Fileset: 5288422 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-10/24/17-12-30.722/ngff/idr0125
Render Image 15923706
fs_contents ['SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-10/24/17-12-30.722/ngff/idr0125/SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015201__2016-04-25T02_38_35-Measurement2.ome.zarr
Plate:11001
- after "Save to All" takes 20 mins to generate thumbnails for first Field of all Wells:
Took 15 mins to repeat "Save to All" thumbnails for field 2.
For the next Plate, compare render
thumbnail generation...
omero --debug=2 render set Plate:11002 screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_renderplate_11002.log 2>/tmp/idr0125_renderplate_11002.err;
In about 2.5 hours of render set Plate:11002
the logs show less than 400 thumbnails generated...
This is 2.4 thumbnails per minute! Even though the logs say got thumbnail in 0.60s
etc.
$ cat /tmp/idr0125_renderplate_11002.err | grep "got thumbnail" | wc
384 1920 14635
$ tail -f /tmp/idr0125_renderplate_11002.err
Updated rendering settings for Image:15920249
Image:15920249 got thumbnail in 0.71s
Updated rendering settings for Image:15920250
Image:15920250 got thumbnail in 0.60s
Updated rendering settings for Image:15920251
Image:15920251 got thumbnail in 0.63s
Updated rendering settings for Image:15920252
Image:15920252 got thumbnail in 0.73s
Updated rendering settings for Image:15920253
Image:15920253 got thumbnail in 0.79s
Now running 5 hours...
$ cat /tmp/idr0125_renderplate_11002.err | grep "got thumbnail" | wc
774 3870 29498
Cancel and try different approach...
Use render copy
...
omero --debug=2 render copy Image:15910651 Plate:11004 > /tmp/idr0125_renderplate_11004.log 2>/tmp/idr0125_renderplate_11004.err;
Completed in less than 5 mins: - mix of Error and success...
$ tail /tmp/idr0125_renderplate_11004.err
Error: Image:15930288
Error: Image:15930289
Error: Image:15930297
Error: Image:15930298
Image:15930281 got thumbnail in 0.96s
Image:15930272 got thumbnail in 0.65s
Image:15930263 got thumbnail in 0.50s
Image:15930254 got thumbnail in 0.57s
Image:15930245 got thumbnail in 0.58s
Image:15930236 got thumbnail in 0.44s
# one row per image...
$ cat /tmp/idr0125_renderplate_11004.err | wc
3457 8075 82259
# Seems we only got 1 success per Well!!
$ cat /tmp/idr0125_renderplate_11004.err | grep "got thumbnail" | wc
384 1920 14592
Hmmm - plate maybe not ready... Same issue as above with only 1 Image per Well being visible
Try different plate... 14:55...
$ omero --debug=2 render copy Image:15910651 Plate:11002 > /tmp/idr0125_copyplate_11002.log 2>/tmp/idr0125_copyplate_11002.err;
This is going much slower... After about 30 mins...
$ cat /tmp/idr0125_copyplate_11002.err | wc
1029 2405 24592
And is still seeing most images are errors:
$ cat /tmp/idr0125_copyplate_11002.err | grep thumb | wc
112 560 4336
This plate took a lot longer to copy rdefs, probably because I'd not updated symlinks on Plate:11004
so it was fast.
For Plate:11002
, took ~90 mins
$ cat /tmp/idr0125_copyplate_11002.err | wc
3457 8075 82611
Again, just 384 thumbs created:
$ cat /tmp/idr0125_copyplate_11002.err | grep "thumb" | wc
384 1920 14944
For the "got thumbnail" images (first image in each Well), the rendering settings result in white images. Other images didn't get rendering settings applied, so they generate thumbs with default settings:
Trying to debug the render copy
using single images...
Copy from first image of Plate:11001
- (see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1778743245) to the first image in Well above...
(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15910651 Image:15919024
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Rendering settings successfully copied to 1 images.
Image:15919024 got thumbnail in 7.27s
This "works" as before when doing whole Plate, but image is still "white".
Copy from same image to Image at 2nd field in Well - or from 1st field in this Well to 2nd field - both FAIL
(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15910651 Image:15919025
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Error: Image:15919025
Rendering settings successfully copied to 0 images.
(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15919024 Image:15919025
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Error: Image:15919025
Rendering settings successfully copied to 0 images.
Copy from 2nd Field to 3rd Field - this "works" ok
(venv3) (base) bash-4.2$ omero --debug=2 render copy Image:15919025 Image:15919026
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Rendering settings successfully copied to 1 images.
Image:15919026 got thumbnail in 3.05s
I notice that the Channel Names
are different between 1st Field of each well: HOECHST 33342, Alexa 488, 488 long, Alexa 568, Alexa 647
and other 8 Fields: 0, 1, 2, 3, 4
(no names).
This probably explains failure to copy settings...
Checking Channels for these Images... For the first Image in a Well, we have 5 different LogicalChannels (with labels):
idr=> select id, logicalchannel from Channel where pixels=(select id from Pixels where Image=15919024);
id | logicalchannel
----------+----------------
49552120 | 16077311
49552119 | 16077310
49552118 | 16077309
49552117 | 16077308
49552116 | 16077307
(5 rows)
But for the 2nd Image in a Well, all the LogicalChannels are the same:
idr=> select id, logicalchannel from Channel where pixels=(select id from Pixels where Image=15919025);
id | logicalchannel
----------+----------------
49552125 | 16077312
49552124 | 16077312
49552123 | 16077312
49552122 | 16077312
49552121 | 16077312
(5 rows)
Set symlinks on newly imported Plates...
for i in {11003..11010}; do
echo "$i"
python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
After updating to https://github.com/ome/omero-cli-render/pull/59/commits/35089403998581db74d26e2f107bdb5b622ae953
$ omero --debug=2 render copy Image:15910651 Plate:11003 > /tmp/idr0125_PR59_copyplate_11003.log 2>/tmp/idr0125_PR59_copyplate_11003.err
Nearly 11 hours...
$ tail /tmp/idr0125_PR59_copyplate_11003.err
..
Image:15925282 got thumbnail in 23.75s
Image:15925265 got thumbnail in 0.10s
Image:15925266 got thumbnail in 24.15s
Image:15925267 got thumbnail in 25.36s
Image:15925268 got thumbnail in 23.04s
Image:15925269 got thumbnail in 23.97s
Image:15925270 got thumbnail in 26.02s
Image:15925271 got thumbnail in 23.07s
Image:15925272 got thumbnail in 24.89s
Image:15925273 got thumbnail in 23.91s
Image:15925256 got thumbnail in 0.10s
Every 9th thumbnail is much faster. These are the First field images that are white (see https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1779561095).
EDIT: still not done, but cancelling now...
$ cat /tmp/idr0125_PR59_copyplate_11003.err | wc
2323 11625 90325
The settings have not been copied properly (at-all?). This is from first Field of A1 in Plate:11001 to the whole Plate:11003.
Previously, using the webclient, "Save to All" saves from an Image to the same Plate, then also updates Thumbnails for the same Field index as that Image, so this wouldn't show differences between Fields.
Testing settings on a new Plate:11004
...
Save rendering settings to an Image at 2nd Field - then "Save to All" in webclient. This applies those settings to all images int the Plate except for First Field.
Try copying settings from First Image in A1 to the same Plate using...
$ screen -r idr0125_symlink2
$ omero --debug=2 render copy Image:15936763 Plate:11005 > /tmp/idr0125_PR59_copyplate_11005.log 2>/tmp/idr0125_PR59_copyplate_11005.err
EDIT: 3 hours - progress
$ cat /tmp/idr0125_PR59_copyplate_11005.err | wc
420 2110 16422
But this is still not working. omero render copy
doesn't apply settings to Plate, even on compatible images:
Since we have issues with OMEZarrReader_performance_allfiles.jar
above, reverted to use
OMEZarrReader-0.3.1-jar-with-dependencies.jar
- restarted server
Removed 18 rows from plates.tsv
to include last-imported plate SQ00015217__2016-05-04T22_08_57-Measurement1
and restarted import...
$ omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231027.log --errs /tmp/idr0125_20231
Trying to view images from Plate:10953
(not previously viewed since updating symlinks is failing:
2023-10-27 13:46:14,419 ERROR [ ome.io.bioformats.BfPixelBuffer] (.Server-13) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml
2023-10-27 13:46:14,419 ERROR [ ome.io.nio.PixelsService] (.Server-13) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2023-10/20/18-59-57.223/SQ00015154__2016-04-14T16_07_53-Measurement1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
at ome.logic.RenderingSettingsImpl.resetDefaults(RenderingSettingsImpl.java:520)
at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:650)
at ome.logic.RenderingSettingsImpl.resetDefaultsInSet(RenderingSettingsImpl.java:1498)
at ome.services.ThumbnailCtx$2.actOnOneGroup(ThumbnailCtx.java:407)
at ome.services.PerGroupActor.actOnByGroup(PerGroupActor.java:92)
at ome.services.ThumbnailCtx.createAndPrepareMissingRenderingSettings(ThumbnailCtx.java:411)
at ome.services.ThumbnailBean.getThumbnailByLongestSideSet(ThumbnailBean.java:1003)
at jdk.internal.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source)
Blitz logs don't look good just now - importing and trying to view too...
Going to restart server...
I can't even view previously viewable idr0125 images now! ResourceError etc....
SQ00015152__2016-06-09T03_29_42-Measurement
Current ZarrReader.jar matches the one used to import those plates: OMEZarrReader-0.3.1-jar-with-dependencies.jar
Start again! Import a few new plates with ZarrReader 0.3.1 and use these to test rendering settings workflow...
Need to edit bulk.yml
to comment out # exclude: "clientpath"
since we are importing from same location as before...
Also reset plates.yml
to full list of plates
Rename Screen to idr0125-way-cellpainting/screenA - Oct 2022
so a new one will be created by import...
omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231030.log --errs /tmp/idr0125_20231030.err
Plate: 11051 imported... update symlinks...
python idr-utils/scripts/managed_repo_symlinks.py Plate:11051 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
fileset_dirs {}
Fileset: 5288471 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2023-10/30/11-46-57.796/
Render Image 16024912
fs_contents ['SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2023-10/30/11-46-57.796/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Apply rendering settings to 1 image.. 16021651
omero render set Image:16021651 screenA/idr0125-screenA-renderdef.yml
Copy from 1 image to plate 16:38...
omero --debug=2 render copy Image:16021651 Plate:11051 > /tmp/idr0125_PR59_copyplate_11051.log 2>/tmp/idr0125_PR59_copyplate_11051.err
EDIT: 22:40 (6 hrs):
$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
1475 7385 57595
$ tail -f /tmp/idr0125_PR59_copyplate_11051.err
Image:16023646 got thumbnail in 13.73s
Image:16023647 got thumbnail in 14.26s
Image:16023648 got thumbnail in 12.48s
Image:16023631 got thumbnail in 13.94s
Image:16023632 got thumbnail in 14.10s
Image:16023633 got thumbnail in 13.19s
Image:16023634 got thumbnail in 13.87s
Image:16023635 got thumbnail in 12.93s
Image:16023636 got thumbnail in 13.69s
Image:16023637 got thumbnail in 14.83s
Batches of thumbnails being updated periodically...
Last one at 22:24
.
$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 57K Oct 30 22:24 /tmp/idr0125_PR59_copyplate_11051.err
$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 65K Oct 30 23:12 /tmp/idr0125_PR59_copyplate_11051.err
$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
1686 8440 65824
$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 73K Oct 31 00:00 /tmp/idr0125_PR59_copyplate_11051.err
$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
1897 9495 74053
25 mins later, no change! Batches updated quite infrequently!
...Done after 13 hours 20 mins
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ cat /tmp/idr0125_PR59_copyplate_11051.err | wc
3458 17295 134893
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls -alh /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 132K Oct 31 06:02 /tmp/idr0125_PR59_copyplate_11051.err
Since there was long waits between setting batch generation of thumbnails (default batch=100), let's simply do 1 thumbnail at a time. There doesn't seem to be any advantage of doing this in batches, since for each batch, the thumbnails are generated 1 at a time.
Update symlinks on a bunch of newly-imported plates (before viewing them)... (import still running)
9:50...
for i in {11052..11058}; do
echo "$i"
python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
...
Fileset: 5288478 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-16/2023-10/31/05-52-01.142/
Render Image 16047490
fs_contents ['SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-16/2023-10/31/05-52-01.142/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr
...slow but all completed OK
Picked first image from Plate: 11052... Updated to https://github.com/ome/omero-cli-render/pull/59/commits/b8fba620a2b882050c1858f3a7d8bee8c8efdede 10:49...
omero --debug=2 render set Image:16025107 screenA/idr0125-screenA-renderdef.yml
$ omero --debug=2 render copy Image:16025107 Plate:11052 > /tmp/idr0125_PR59_copyplate_11052.log 2>/tmp/idr0125_PR59_copyplate_11052.err
...11:05 - still nothing in /tmp/idr0125_PR59_copyplate_11052.err
!
At 12:21... - no updates to the err since 11:39!
(base) bash-4.2$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
209 1055 8221
(base) bash-4.2$
(base) bash-4.2$ tail -f /tmp/idr0125_PR59_copyplate_11052.err
Image:16028356 got thumbnail in 15.48s
Image:16028357 got thumbnail in 13.78s
Image:16028358 got thumbnail in 12.15s
Image:16028359 got thumbnail in 13.84s
Image:16028360 got thumbnail in 11.72s
Image:16028361 got thumbnail in 14.74s
Image:16028362 got thumbnail in 15.51s
Image:16028363 got thumbnail in 13.39s
Image:16028364 got thumbnail in 12.54s
Image:16028347 got thumbnail in 18.88s
(base) bash-4.2$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 8.1K Oct 31 11:39 /tmp/idr0125_PR59_copyplate_11052.err
At 15:50... - no .err update since 15:01.
$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
1053 5275 41137
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 41K Oct 31 15:01 /tmp/idr0125_PR59_copyplate_11052.err
23:00...
$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
2952 14770 115197
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 113K Oct 31 22:35 /tmp/idr0125_PR59_copyplate_11052.err
Done in 13 hours 40 mins - slightly slower than before!
$ cat /tmp/idr0125_PR59_copyplate_11052.err | wc
3458 17295 134893
$ ls -alh /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 1 00:32 /tmp/idr0125_PR59_copyplate_11052.err
Image:16025108 got thumbnail in 14.12s
Image:16025109 got thumbnail in 13.35s
Image:16025110 got thumbnail in 13.87s
Image:16025111 got thumbnail in 14.31s
Image:16025112 got thumbnail in 12.92s
Image:16025113 got thumbnail in 14.49s
Image:16025114 got thumbnail in 24.59s
Image:16025115 got thumbnail in 14.69s
Image:16025116 got thumbnail in 14.56s
13 hours (3456 images) is 13.5 secs per image - which fits with the logs. Just strange that the logs aren't updated very frequently.
13 hours to get thumbnails for a plate is better than previous rates of >2days per plate: https://idr-redmine.openmicroscopy.org/issues/228?issue_count=1&issue_position=1#note-109
Need to know if we can copy settings from an Image in one plate to a different Plate (didn't work before with messed-up channels). Also try applying settings in multiple treads as before (see redmine link) in multiple terminal screens.
First, update symlinks for newly imported plates... (import still running)...
for i in {11059..11084}; do
echo "$i"
python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
...
After an 1hr 20mins, done 24/27 links... - At this rate, it will take 7.5 hours to do all 136 Plates!
...
Fileset: 5288502 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/00-52-58.990/
Render Image 16129507
fs_contents ['SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/00-52-58.990/SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015099__2016-06-08T13_48_06-Measurement1.ome.zarr
11083
fileset_dirs {}
Fileset: 5288503 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2023-11/03/03-34-08.818/
Render Image 16134402
...
..Also want to compare whether omero render copy Image:ID Plate:ID
(uses settingsService to set whole plate at once) is faster than `omero render set Plate:ID rendering.yml (saves settings to one image at at time).
Let's use a batch of 5 plates for each...
Pick an image from first Plate:11059, apply settings to that Image, then use it to apply settings to 5 Plates (including it's own plate)...
omero --debug=2 render set Image: 16049299 screenA/idr0125-screenA-renderdef.yml
for i in {11059..11063}; do
echo "$i"
omero --debug=2 render copy Image:16049299 Plate:$i > /tmp/idr0125_PR59_copyplate_$i.log 2>/tmp/idr0125_PR59_copyplate_$i.err
done
In another screen..
for i in {11064..11068}; do
echo "$i"
omero --debug=2 render set Plate:$i screenA/idr0125-screenA-renderdef.yml > /tmp/idr0125_setplate_$i.log 2>/tmp/idr0125_PR59_setplate_$i.err
done
EDIT: few days later... For copy plate...
$ ls -alh /tmp/idr0125_PR59_cop*
-rw-r--r--. 1 omero-server omero-server 132K Oct 31 06:02 /tmp/idr0125_PR59_copyplate_11051.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 1 00:32 /tmp/idr0125_PR59_copyplate_11052.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 4 01:19 /tmp/idr0125_PR59_copyplate_11059.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 4 14:16 /tmp/idr0125_PR59_copyplate_11060.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 5 02:57 /tmp/idr0125_PR59_copyplate_11061.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 5 15:44 /tmp/idr0125_PR59_copyplate_11062.err
-rw-r--r--. 1 omero-server omero-server 132K Nov 6 04:17 /tmp/idr0125_PR59_copyplate_11063.err
Thumbnails look good for plate:11063. From logs it looks like thumbnails took 13 hours per Plate, and copying from an Image on one plate to a different Plate worked OK! 👍
For set...
$ ls -alh /tmp/idr0125_PR59_set* | grep err
-rw-r--r--. 1 omero-server omero-server 284K Nov 4 11:12 /tmp/idr0125_PR59_setplate_11064.err
-rw-r--r--. 1 omero-server omero-server 284K Nov 5 09:39 /tmp/idr0125_PR59_setplate_11065.err
-rw-r--r--. 1 omero-server omero-server 284K Nov 6 08:00 /tmp/idr0125_PR59_setplate_11066.err
-rw-r--r--. 1 omero-server omero-server 281K Nov 7 06:51 /tmp/idr0125_PR59_setplate_11067.err
-rw-r--r--. 1 omero-server omero-server 1.1K Nov 7 06:51 /tmp/idr0125_PR59_setplate_11068.err
This looks like it took about 22 hours for each plate and it was only midway through the penultimate plate when I restarted the server below on the 7th Nov.
Will restart thumbnailing using copy
command below...
It looks like the most recent OMEZarrReader fixes had addressed the channel issues seen above.
Stopped import (after 67 Plates) - last plate imported was SQ00015138__2016-06-11T17_59_08-Measurement1
.
Updated ZarrReader to today's merge build https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/673/console
Deleted 67 lines from idr0125-plates.tsv
and restarted import...
omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231107.log --errs /tmp/idr0125_20231107.err
Symlinks for newly imported plates... 10:15...
for i in {11085..11117}; do
echo "$i"
python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
EDIT: failed with DatabaseBusyException
since I am importing and viewing in webcilent at the same time.
Re-ran...
After Plate:11117 (last plate imported before server restart), IDs jump to Plate:11151...
Plates that still need thumbnailing...
11053-11058 11067-11117 11151-...
On idr0125-pilot, going to test import of METADATA.ome.xml and then use mkngff to replace the fileset with full NGFF plate mounted on s3.
Running mkngff to replace OME.METADATA.ome.xml single-file Fileset with NGFF plate...