Closed will-moore closed 12 months ago
Looks like full plate downloaded
$ find ./ -name .zattrs | wc
3842 3842 270564
python add_downsampling.py SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr 8
Then upload to bucket..
$ /home/wmoore/mc cp -r SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr uk1s3/idr0125/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr
Looks like I got the mc cp
command wrong - added an extra directory level:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr
I must have run python add_downsampling.py SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr 8
twice, since each image has 2 extra resolutions.
Repeat with a new Plate...
mkdir SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
aws s3 sync --no-sign-request s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
time python add_downsampling.py SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr 8
...
real 40m16.263s
user 31m5.552s
sys 3m26.322s
/home/wmoore/mc cp -r SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr uk1s3/idr0125
Validate that this plate has a single 8-factor downsampling... https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Views well in Vizarr:
Zoom-in gives good resolutions of thumbnails (plate view):
Single image looks good too (no issues with 8 x factor scaling):
To test import and thumbnailing of this Plate above, I'll use released ZarrReader on idr0125-pilot (to avoid black-image and import-failure issues with ZarrReader performance improvements branch)
Copy local OMEZarrReader jar to idr0125-pilot...
rsync -rvP --progress ~/Downloads/OMEZarrReader-0.3.1-jar-with-dependencies.jar idr-pilot.openmicroscopy.org:/home/wmoore
ssh -A idr-pilot.openmicroscopy.org
rsync -rvP --progress OMEZarrReader-0.3.1-jar-with-dependencies.jar idr0125-omeroreadwrite:/home/wmoore
ssh idr0125-omeroreadwrite
sudo cp OMEZarrReader-0.3.1-jar-with-dependencies.jar OMEZarrReader.jar
sudo chown omero-server OMEZarrReader.jar
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
sudo service omero-server restart
Going to try....
1) Use a tried-and-tested regular import of plate-without-chunks, then update symlinks to include chunks. 2) Also try import of METADATA.ome.xml and use mkngff to create Fileset from full Plate.
screen -S idr0125_aws_sync
cd idr0125
mkdir SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
aws s3 sync --no-sign-request --exclude '*' --include "*/.z*" --include "*.xml" --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
# copied to /ngff/idr0125 and chown to omero-server, then...
$ omero import --transfer=ln_s --depth=100 --name=SQ00014812__2016-05-23T20_44_31-Measurement1 --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125_20230914.log --errs /tmp/idr0125_20230914.err
...
2023-09-14 14:52:03,766 16610619 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 52235482
2023-09-14 14:52:03,791 16610644 [l.Client-5] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 52235482
2023-09-14 14:52:03,832 16610685 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 52235482
2023-09-14 14:52:03,911 16610764 [l.Client-5] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 52235482
2023-09-14 14:52:05,000 16611853 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287574
==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 4:36:34.647
http://localhost:1080/webclient/?show=plate-10601
On idr0125-pilot, mount s3 bucket...
sudo mkdir /idr0125 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0125 /idr0125
$ ls /idr0125
SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr
Create symlink
sudo -u omero-server -s
rm -rf /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-09/14/10-15-31.104/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
ln -s /idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-09/14/10-15-31.104/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
This allowed chunks to be viewed and on "Save to all" thumbnails were generated quickly - about a minute for the whole plate:
cell-paininting data now has 2nd, lower resolution, e.g. https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/
Since the current data have only 1
multiscales
resolution, generation of thumbnails in IDR takes a long time as the full-scale images must be retrieved to generate thumbnails.The script at https://github.com/IDR/idr0125-way-cellpainting/blob/ffe378766685f418774f43ee5b78913ac57c679a/scripts/add_downsampling.py is in development as a way of adding down-sampled resolutions.
To test, lets download a full plate... On
idr-ftp
server...