idr-testing May 2024

[will-moore]

Steps needed on idr-next for NGFF upgrade. NB: current checklist is for actions on idr-testing (newly redeployed on 21st May 2024)

Detailed workflow is at https://github.com/IDR/mkngff_upgrade_scripts but this is an outline, also includes study-specific jobs:

Manual Software updates (should be part of the original deployment for idr-next):

• [x] Update ZarrReader if needed to include recent work
• [x] Install mkngff in venv, including recent PR branches (if not yet merged)
• [x] Install latest iviewer 0.14.0

NGFF and other udpates:

• [x] Add bfoptions file to skip_wells for all idr0009 Plates
• [x] idr0070: remove annotations, delete svs images, re-import and re-annotate as described
• [x] Create an OME-NGFF Tag and add the Screens and Projects
• [x] As omero-server, clone https://github.com/IDR/mkngff_upgrade_scripts
• [x] idr0015 - delete one duplicate plate named TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01
• [x] Get $SECRET, update all sql commands with it and run them (see https://github.com/IDR/mkngff_upgrade_scripts)
• [x] Run mkngff symlink on all studies, including bfoptions creation
• [x] Update bfoptions files for idr0004 plates P115 and P124
• ~Regenerate thumbnails for idr0015 plate https://idr.openmicroscopy.org/webclient/?show=plate-4653~
• [ ] Generate thumbnails for idr0012 Plate 130-16, by going to an Image (e.g. http://localhost:1080/webclient/?show=image-3063425 and "Save To All"
• [ ] Bio-Formats cache regeneration
• [ ] Validation? E.g. https://github.com/IDR/idr-utils/pull/55 on some data?
• [x] Move idr00XX.csv and sql files from https://github.com/IDR/idr-utils/pull/56 into separate repos, then review/merge the PR.

[will-moore]

Missed memo files for idr0009-ScreenB

Now on idr-next...

omeroreadwrite:

/opt/omero/server/OMERO.server/bin/omero hql --limit -1 --ids-only --style csv 'select MIN(field.image.id) FROM WellSample AS field where field.well.plate.id in (select link.child.id from ScreenPlateLink as link where link.parent.id=803) GROUP BY field.well.plate' > idr0009B_plates.txt

cut -d ',' -f2 idr0009B_plates.txt | sed -e 's/^/Image:/' > idr0009B_plate_ids.txt

[wmoore@prod122-omeroreadwrite ~]$ screen -dmS cache parallel --eta -a idr0009B_plate_ids.txt --results /tmp/ngff_cache_20240701_idr0009B/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

20 mins later...

[wmoore@prod122-omeroreadwrite ~]$ grep "ok:" /tmp/ngff_cache_20240701_idr0009B/1/**/*
...
/tmp/ngff_cache_20240701_idr0009B/1/Image:1337225/stdout:ok: Pixels:1337225 Image:1337225 96.57122135162354 
/tmp/ngff_cache_20240701_idr0009B/1/Image:1337609/stdout:ok: Pixels:1337609 Image:1337609 103.64140939712524 
/tmp/ngff_cache_20240701_idr0009B/1/Image:1337993/stdout:ok: Pixels:1337993 Image:1337993 131.27101874351501 
/tmp/ngff_cache_20240701_idr0009B/1/Image:1338377/stdout:ok: Pixels:1338377 Image:1338377 84.42854690551758 
/tmp/ngff_cache_20240701_idr0009B/1/Image:1339145/stdout:ok: Pixels:1339145 Image:1339145 0.20086431503295898 
/tmp/ngff_cache_20240701_idr0009B/1/Image:1481751/stdout:ok: Pixels:1481751 Image:1481751 0.836538553237915 

[wmoore@prod122-omeroreadwrite ~]$ grep "ok:" /tmp/ngff_cache_20240701_idr0009B/1/**/* | wc
     69     276    7557

70 Plates - 1 ResourceError checked OK in webclient.