idr-next steps for NGFF upgrade

[will-moore]

Steps needed on idr-next for NGFF upgrade. NB: current checklist is for actions on idr-testing (newly redeployed on 10th Oct)

Detailed workflow is at https://github.com/IDR/omero-mkngff/issues/2 but this is an outline, also includes study-specific jobs:

• [ ] Add bfoptions file to skip_wells for all idr0009 Plates
• [ ] idr0070: remove annotations, delete svs images, re-import and re-annotate as described
• [ ] Create an OME-NGFF Tag and add the Screens and Projects
• [ ] Update ZarrReader if needed to include recent work
• [ ] Install mkngff in venv, including recent PR branches (if not yet merged)
• [ ] As omero-server, clone https://github.com/IDR/mkngff_upgrade_scripts
• [ ] idr0015 - delete one duplicate plate named TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01
• [ ] Get $SECRET, update all sql commands with it and run them (see https://github.com/IDR/mkngff_upgrade_scripts)
• [ ] Run mkngff symlink on all studies, including bfoptions creation
• [ ] Update bfoptions files for idr0004 plates P115 and P124
• ~Regenerate thumbnails for idr0015 plate https://idr.openmicroscopy.org/webclient/?show=plate-4653~
• [ ] Generate thumbnails for idr0012 Plate 130-16, by going to an Image (e.g. http://localhost:1080/webclient/?show=image-3063425 and "Save To All"
• [ ] Bio-Formats cache regeneration
• [ ] Validation? E.g. https://github.com/IDR/idr-utils/pull/55 on some data?
• [x] Move idr00XX.csv and sql files from https://github.com/IDR/idr-utils/pull/56 into separate repos, then review/merge the PR.

[will-moore]

Delete on idr-testing...

(venv3) [wmoore@test120-omeroreadwrite ~]$ omero delete Plate:4801 --report > delete_report.log
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public

omero.cmd.Delete2 Plate:4801 ok
Steps: 6
Elapsed time: 129.884 secs.
Flags: []
Deleted objects
  CommentAnnotation:1502861
  FilesetAnnotationLink:20901
  MapAnnotation:8368853
  PlateAnnotationLink:3493
  Channel:4731551-4733530
  Image:1958101-1958496
  LogicalChannel:38501
  OriginalFile:6948775-6988376
  Pixels:1958101-1958496
  ChannelBinding:4269391-4271370
  QuantumDef:1750202-1750597
  RenderingDef:1750202-1750597
  Thumbnail:1738526-1738921,4222134,4222136-4222139,4222144,4222146,4222165-
....
4222496,4222499,4222500,4255156-4255161,4255163-4255199,4255203
  Fileset:21101
  FilesetEntry:6810401-6848218,6848228,6848241
.....
6861554,6861559,6861564,6861569,6861574,6861579,6861584,6861590
  FilesetJobLink:99401-99405
  IndexingJob:106219
  JobOriginalFileLink:33515
  MetadataImportJob:106216
  PixelDataJob:106217
  ThumbnailGenerationJob:106218
  UploadJob:106215
  Plate:4801
  ScreenPlateLink:5001
  Well:1056601-1056996
  WellSample:1762101-1762496
  StatsInfo:3976732-3978711

[will-moore]

sudo -u root -s
source /opt/omero/server/venv3/bin/activate
pip install 'omero-mkngff @ git+https://github.com/IDR/omero-mkngff@main'

[will-moore]

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr-utils/
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git fetch will
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git checkout will/filesets_to_swap

[will-moore]

Latest Bio-Formats push: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/643/console

2023-10-10 00:05:33,744 [   scc.merge] INFO  Repository: ome/ZarrReader
2023-10-10 00:05:33,744 [   scc.merge] INFO  Excluded PRs:
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO  Already up-to-date.
2023-10-10 00:05:33,744 [   scc.merge] INFO  
2023-10-10 00:05:33,744 [   scc.merge] INFO  Merged PRs:
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 62 joshmoore 'Update README.md'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 64 dgault 'ZarrReader performance improvements'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 65 dgault 'Bumping Bio-Formats version to 7.0.0'

Download from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Run on omeroreadwrite, omeroreadonly-1 - 4...

sudo -u omero-server -s
cd
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p643_b565.jar
rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar && rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/client/ && cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/server/
# as wmoore
sudo service omero-server restart

[will-moore]

On omeroreadwrite...

• as omero-server, screen -S mkngff... - get SECRET
• as wmoore...

cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

for dir in $(ls -d */); do
  cd $dir
  for i in $(ls); do sudo sed -i 's/SECRETUUID/1def0359-65xyz-443f-b14a-69f04f569f4d/g' $i; done
  cd ../;
done

• as omero-server, screen -S mkngff...

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
UPDATE 95
BEGIN
 mkngff_fileset 
----------------
        6312382
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr -> /bia-integrator-data/S-BIAD867/00d88a93-8d21-4a50-b8b5-60f11bcae0d3/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr
UPDATE 98
BEGIN
 mkngff_fileset 
----------------
        6312383
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864
...

[will-moore]

Ran for all studies up to idr0016 yesterday... Today...

• idr0025 9:47
• idr0026 9:48
• idr0033 9:52
• idr0036 10:12
• idr0051 10:23
• idr0054 10:24
• idr0090 10:25
• idr0091 10:29 (done at 10:50)

error from idr0054 because idr0054.csv has whitespace in rows - but shouldn't cause any errors as the correct IDs were still found? /opt/omero/server/venv3/bin/omero mkngff symlink: error: argument fileset_id: invalid int value: 'idr0054/Tonsil'

[will-moore]

Saved memo times for the first Fileset in each study...

e.g.

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.4 | grep -A 2 "81ba6c774fdd"
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/..zattrs.bfmemo (91291 bytes)
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1697033824100] time[17840] tag[loci.formats.Memoizer.setId]
2023-10-11 14:17:21,941 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/.zattrs Series: 0

• idr0004 139197ms (2.3 mins)
• idr0010 694251ms (11.5 mins)
• idr0011 68187ms (1.1 mins)
• idr0012 1248415ms (20.8 mins)
• idr0013 1125667ms (18.7 mins)
• idr0015 1163426ms (19.3 mins)
• idr0016 3221519ms (53.7 mins)
• idr0025 543319ms (9.1 mins)
• idr0026 22904ms (0.4 mins)
• idr0033 7688852ms (128 mins)
• idr0035 451787ms (7.5 mins)
• idr0036 7228099ms (120 mins)
• idr0051 578885ms (9.6 mins)
• idr0054 2476ms (0.04 mins)
• idr0090 6702958ms (111 mins)
• idr0091 17840ms (0.3 mins)

[will-moore]

As omero-server... With https://github.com/IDR/omero-mkngff/pull/13/commits/7dc8222c0822c7a7f422cb8c46e9f7a9a53c9541 installed And the ZarrReader that supports bfoptions updated https://github.com/ome/ZarrReader/pull/64#issuecomment-1787136526

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff bfoptions /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr.bfoptions

This now allows viewing of all idr0004. Repeat for others...

All completed!

[will-moore]

Using https://github.com/IDR/idr-utils/pull/55, let's validate a bunch of Plates etc.. 1st plate from each study...

First plate IDs from idr0004 etc.

omero login
cd /uod/idr/metadata/idr-utils/scripts

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
python check_pixels.py Plate:$i /tmp/check_pix_$i.log --max-planes=1; done
...

tail -f /tmp/check_pix_4287.log 
73/672 Check Image:1811320 HT01 [Well M21, Field 1]
74/672 Check Image:1811321 HT01 [Well M21, Field 2]
Error: Mismatch for Image: 1811321 at plane (z, c, t): (0, 0, 0)
75/672 Check Image:1811322 HT01 [Well P5, Field 1]
76/672 Check Image:1811323 HT01 [Well P5, Field 2]
Error: Mismatch for Image: 1811323 at plane (z, c, t): (0, 0, 0)
77/672 Check Image:1811324 HT01 [Well M15, Field 1]
78/672 Check Image:1811325 HT01 [Well M15, Field 2]
Error: Mismatch for Image: 1811325 at plane (z, c, t): (0, 0, 0)
79/672 Check Image:1811326 HT01 [Well E20, Field 1]
80/672 Check Image:1811327 HT01 [Well E20, Field 2]
Error: Mismatch for Image: 1811327 at plane (z, c, t): (0, 0, 0)

[will-moore]

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
grep -c Mismatch /tmp/check_pix_$i.log; done

Plate:1751 (idr0004 1-field)
0
Plate:4501 (idr0010 1-field)
0
Plate:5387 (idr0011 1-field)
0
Plate:4287 (idr0012 2-fields)
336
Plate:3451 (idr0013 1-field)
0
Plate:4295 (idr0015 1-field)
0
Plate:6208 (idr0016 6-fields) omero-cli-zarr
0
Plate:6001 (idr0025 4-fields)
288
Plate:5958 (idr0033 9-fields)
3072
Plate:6301 (id0035 4-fields)
180
Plate:4403 (idr0036 9-fields) omero-cli-zarr
0
Plate:9301. (idr0090 32-fields) omero-cli-zarr
0

The Errors come from all Fields except the first field in multi-field Plates. Only with bioformats2raw Plates. No issues with omero-cli-zarr. All the error counts for the 4 plates above exactly match Wells x (fields - 1). When viewing images in these plates, all appear fine.

For a single Image - on idr-testing, connect to localhost or idr.openmicroscopy.org. Compare same images...

http://localhost:1080/webclient/?show=image-1811321 https://idr.openmicroscopy.org/webclient/?show=image-1811321

>>> conn = BlitzGateway("public", "public", host="localhost", port=4064)
>>> client = omero.client("idr.openmicroscopy.org", 4064)
>>> sf = client.createSession("public", "public")
>>> conn2 = BlitzGateway(client_obj=client)
>>> image = conn.getObject('Image', 1811321)
>>> plane = image.getPrimaryPixels().getPlane(0, 0, 0)
>>> image2 = conn2.getObject('Image', 1811321)
>>> plane2 = image2.getPrimaryPixels().getPlane(0, 0, 0)
>>> plane
array([[1596, 1602, 1625, ..., 1738, 1738,  586],
       [1634, 1641, 1614, ..., 1689, 1758,  586],
       [1746, 1717, 1723, ..., 1631, 1654,  586],
       ...,
       [1288, 1351, 1375, ..., 1243, 1275,  602],
       [1276, 1272, 1290, ..., 1225, 1250,  600],
       [ 618,  620,  618, ...,  594,  596,  600]], dtype=uint16)
>>> plane2
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype=uint16)

Pixel values are different!

With the same rendering settings, these images do appear a little different:

Need to check the raw NGFF data...

[will-moore]

>>> url = "https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD845/409227a9-2f3f-421c-8086-b14253283e42/409227a9-2f3f-421c-8086-b14253283e42.zarr/M/21/1/"
>>> from ome_zarr.io import parse_url
>>> from ome_zarr.reader import Reader
>>> store = parse_url(url, mode="r").store
>>> reader = Reader(parse_url(url))
>>> nodes = list(reader())
>>> len(nodes)
1
>>> image_node = nodes[0]
>>> dask_data = image_node.data
>>> data = dask_data[0].compute()
>>> data.shape
(1, 3, 1, 1024, 1344)
>>> data[0,0,0]
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype='>u2')

These pixel values appear identical to those loaded from idr.openmicroscopy.org, so it looks like the NGFF data is correct and the differences we're seeing on idr-testing are likely due to the ZarrReader.

[will-moore]

Updated ZarrReader on idr-testing - see https://github.com/ome/ZarrReader/pull/64#issuecomment-1795030443

Generating memo file for previously un-viewed HT03 Plate:4289 from idr0012: http://localhost:1080/webclient/?show=image-1813262 ...

[will-moore]

python check_pixels.py Plate:4289 /tmp/check_pix_4289.log

No images have pixel mismatch (checking all 3 planes for all images). Issue above is fixed!

Known issue - all the C row of images have sizeC=1

$ grep -c Mismatch /tmp/check_pix_4289.log
0

grep  Error /tmp/check_pix_4289.log
...
Error: Image:1813207 Z '1' greater than sizeZ '1'.
Error: Image:1813212 Z '1' greater than sizeZ '1'.
Error: Image:1813213 Z '1' greater than sizeZ '1'.

grep -c Error /tmp/check_pix_4289.log
42

[will-moore]

Since images without chunks didn't import above, test import of idr0054 image from https://github.com/ome/ZarrReader/pull/64#issuecomment-1768612367

$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_20231106.log --errs /tmp/idr0054_20231106.err

...
-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-11-06 21:40:04,010 9144       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-11-06 21:40:04,340 9474       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-11-06 21:40:04,466 9600       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61520472
2023-11-06 21:40:05,084 10218      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61520472
2023-11-06 21:40:12,432 17566      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61520472
2023-11-06 21:40:12,959 18093      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61520472
2023-11-06 21:40:13,001 18135      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61520472
2023-11-06 21:40:13,042 18176      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61520472
2023-11-06 21:40:13,165 18299      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314645

==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:15.205

So this still works 👍

[will-moore]

idr-next (prod120) steps...

• Install mkngff from latest bfoptions branch...

ssh idr-next.openmicroscopy.org -At ssh omeroreadwrite -A
[wmoore@prod120-omeroreadwrite ~]$ sudo -u root -s
[root@prod120-omeroreadwrite wmoore]# source /opt/omero/server/venv3/bin/activate
(venv3) [root@prod120-omeroreadwrite wmoore]# pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@bfoptions'

• Before running sql scripts on idr-next, let's remove all the known Filesets that are not ready (being re-submitted to EBI)... 18 removed from csv files... https://github.com/IDR/idr-utils/pull/56/commits/125c4e559d5020dacbf359bc2bf7694dd86fb708

  $ git diff | grep '\-idr'
  -idr0004/P132.ome.zarr,S-BIAD867/3df515f9-75c8-4cc4-8c29-480ae9817880,13125
  -idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/0e5e1147-449f-4a89-96c6-3a3c85f8c1a7,21851
  -idr0011/Plate13-Red-B.ome.zarr,S-BIAD866/36e53731-585a-4b73-9500-8e8b087b1022,21842
  -idr0011/Plate3-Yellow-B.ome.zarr,S-BIAD866/8ba28efa-377f-4602-8c09-41737075ff08,21789
  -idr0011/Plate11-Blue-A.ome.zarr,S-BIAD866/bd01548b-da85-4a66-8422-7b16995fabba,21834
  -idr0013/LT0080_37.ome.zarr,S-BIAD865/8387705b-16bf-4b14-8884-426b0c16dfff,18655
  -idr0013/LT0066_23.ome.zarr,S-BIAD865/dab29e5a-d36f-430a-a9ff-7a1d6e4ce299,18568
  -idr0013/LT0103_13.ome.zarr,S-BIAD865/df947dfe-ed8f-4dda-a20a-fb9f3a717b47,18728
  -idr0015/TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr,S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f,21118
  -idr0015/TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr,S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53,21160
  -idr0015/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr,S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c,21112
  -idr0016/25585.ome.zarr,S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f,21365
  -idr0016/24634.ome.zarr,S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84,21278
  -idr0016/24667.ome.zarr,S-BIAD851/2c49b893-ec6d-4329-9cc3-569b820075f2,23603
  -idr0016/25909.ome.zarr,S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015,21476
  -idr0016/25592.ome.zarr,S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5,21369
  -idr0016/25675.ome.zarr,S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe,21385
  -idr0016/25740.ome.zarr,S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e,21454

  NB: Also 1 plate from idr0011 that was never created

  idr0011/LT0080_27

• Checkout idr-utils filesets_to_swap branch (NB: this will eventually be moved to a separate repo and archived, instead of being merged into idr-utils)...

[wmoore@prod120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr-utils/
[wmoore@prod120-omeroreadwrite idr-utils]$ sudo -Es git fetch will
[wmoore@prod120-omeroreadwrite idr-utils]$ sudo -Es git checkout will/filesets_to_swap
...
HEAD is now at 125c4e5... Remove Filesets from ngff_filesets/ csv files if not ready

• Delete duplicate plate on idr0015: Log looks exactly the same as above: https://github.com/IDR/idr-metadata/issues/675#issuecomment-1755124242

(venv3) bash-4.2$ omero delete Plate:4801 --report > /tmp/idr0015_delete_report.log
...
omero.cmd.Delete2 Plate:4801 ok
Steps: 6
Elapsed time: 133.92 secs.
Flags: []
Deleted objects
  CommentAnnotation:1502861
  FilesetAnnotationLink:20901
  MapAnnotation:8368853
  PlateAnnotationLink:3493
  Channel:4731551-4733530
  Image:1958101-1958496
...

• Run sql scripts... as omero-server user...

(venv3) bash-4.2$ omero config get | grep omero.db.host
(venv3) bash-4.2$ export DBHOST=***
(venv3) bash-4.2$ omero config get --show-password | grep omero.db.pass
omero.db.pass=*****
(venv3) bash-4.2$ export PGPASSWORD=****
***

as wmoore - repeat for all studiess

cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets/idr0004
for i in $(ls); do sudo sed -i 's/SECRETUUID/****/g' $i; done

as omero-server in screen -S mkngff

$ omero mkngff setup > setup.sql
$ psql -U omero -d idr -h $DBHOST -f setup.sql 
CREATE FUNCTION

$ export IDRID=idr0004
$ cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
done

gff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done^C
(venv3) bash-4.2$
(venv3) bash-4.2$
(venv3) bash-4.2$ 
(venv3) bash-4.2$ omero login
Previous session expired for demo on localhost:4064
Server: [localhost:4064]
Username: [demo]
Password:
Created session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
(venv3) bash-4.2$ 
(venv3) bash-4.2$ pwd
/uod/idr/metadata/idr-utils/scripts/ngff_filesets
(venv3) bash-4.2$ 
(venv3) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
>   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done
UPDATE 95
BEGIN
 mkngff_fileset 
----------------
        6312382
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr -> /bia-integrator-data/S-BIAD867/00d88a93-8d21-4a50-b8b5-60f11bcae0d3/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr.bfoptions
UPDATE 98
BEGIN
 mkngff_fileset 
----------------
        6312383
(1 row)

COMMIT
...

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr -> /bia-integrator-data/S-BIAD867/f791de8d-cd01-4303-8b08-67cbbbb45b64/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr.bfoptions
UPDATE 88
BEGIN
 mkngff_fileset 
----------------
        6312426
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr -> /bia-integrator-data/S-BIAD867/fcc7c91a-f0e6-43f4-93a6-220e9224eda5/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr.bfoptions

• manually updated bfoptions for idr0004 plates P115 and P124 to remove read_fast option

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions
vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-57-40.271_mkngff/ee8872c8-e4b1-41fa-aa4f-a9e3e200c540.zarr.bfoptions

• Repeat for:
  • idr0010
  • ...
  • idr0036...

(venv3) bash-4.2$ export IDRID=idr0036
(venv3) bash-4.2$ 
(venv3) bash-4.2$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]');   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql";   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions; done
UPDATE 3456
BEGIN
psql:idr0036/21190.sql:21533: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
^CCancelled
UPDATE 3456
BEGIN
psql:idr0036/21183.sql:21533: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK

Ooops - missed SECRET key for idr0036... Cancelled after 1st ROLLBACK above. Re-run that one command:

omero mkngff symlink /data/OMERO/ManagedRepository 21190 "/bia-integrator-data/S-BIAD855/02dc024f-2228-4d19-923e-0d55c1224ba3/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr" --bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr -> /bia-integrator-data/S-BIAD855/02dc024f-2228-4d19-923e-0d55c1224ba3/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr.bfoptions

Need to re-run psql commands ONLY, after updating SECRET.. looks god...

$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
done

UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314042
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314043
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314044
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        631404
...

ALL studies done!!!

[will-moore]

Memo file generation progress (running since 9th Dec - Now 11th Dec): 853 Filesets done... 9 errors so far... Known issues, waiting on EBI resubmission...

EDIT: updated 14th Dec after stopped running

[wmoore@prod120-proxy ~]$ ls /tmp/cache-20231209/1/ | wc
   1067    1067   14892

[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/*
/tmp/cache-20231209/1/Image:1556033/stdout:fail: 1556033 1.7507498264312744 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1573071/stdout:fail: 1573071 1.6358556747436523 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1600787/stdout:fail: 1600787 1.5452053546905518 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1949093/stdout:fail: 1949093 0.49617934226989746 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1962455/stdout:fail: 1962455 2.030329465866089 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1964831/stdout:fail: 1964831 1.3607375621795654 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1972041/stdout:fail: 1972041 1.2656726837158203 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2131185/stdout:fail: 2131185 0.8565192222595215 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2330212/stdout:fail: 2330212 1.0435378551483154 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2340843/stdout:fail: 2340843 1.059697151184082 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2376573/stdout:fail: 2376573 0.600783109664917 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2435591/stdout:fail: 2435591 0.6575438976287842 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2486279/stdout:fail: 2486279 1.1877374649047852 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2592263/stdout:fill: 2592263 56292.620924949646 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2594567/stdout:fill: 2594567 46556.722032547 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2596871/stdout:fill: 2596871 43117.009229421616 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2599175/stdout:fill: 2599175 41285.22442698479 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2601479/stdout:fill: 2601479 41891.6873960495 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2603783/stdout:fill: 2603783 41889.01078701019 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2608391/stdout:fill: 2608391 41466.79130220413 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2668193/stdout:fail: 2668193 3547.1275115013123 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2677559/stdout:fail: 2677559 4.968778610229492 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2678921/stdout:fail: 2678921 1.9497594833374023 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2680583/stdout:fail: 2680583 1.4392280578613281 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:797729/stdout:fail: 797729 0.8322896957397461 exception ::omero::ResourceError

1556033, 1573071, 1600787 (idr0013)

1949093 is idr0015 broken plate.

1962455, 1964831, 1972041 (idr0015) are actually all ready to go. Ran them https://github.com/IDR/idr-metadata/issues/645#issuecomment-1849888173

2131185, 2330212, 2340843, 2376573, 2435591, 2486279 are idr0016 - all known

New errors from idr0016: 2592263 Plate name 26071 Data looks valid 2594567 data valid 2596871 validator 2599175 2601479 2603783 2608391 2668193 2677559 2678921 2680583

797729 is idr0004

13th Dec: 9:00...

[wmoore@prod120-proxy ~]$  ls /tmp/cache-20231209/1/ | wc
    992     992   13842
[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/* | wc
     14      70    1522

14th Dec (after memo file generation was paused and restarted with different log location below) Checking old logs...

[wmoore@prod120-proxy ~]$ ls /tmp/cache-20231209/1/ | wc
   1067    1067   14892
[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/* | wc
     25     125    2712

Update the list above...

[will-moore]

Seb "NGFF memo file regeneration process restarted earlier this morning (results under /tmp/cache-20231214)"

[sbesson@prod120-proxy ~]$ grep fail /tmp/cache-20231214/1/*/* | wc
     14      61    1402
[sbesson@prod120-proxy ~]$ grep ok /tmp/cache-20231214/1/*/* | wc
   1043    3129   77501

Some fails are Authentication failed. All others are know errors (idr0015 3 plates now fixed).

[wmoore@prod120-proxy ~]$ grep fail /tmp/cache-20231214/1/*/*
/tmp/cache-20231214/1/Image:1556033/stdout:fail: 1556033 0.3516662120819092 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1573071/stdout:fail: 1573071 0.1804647445678711 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1600787/stdout:fail: 1600787 0.14262938499450684 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1949093/stdout:fail: 1949093 0.06853318214416504 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2131185/stdout:fail: 2131185 0.2601206302642822 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2330212/stdout:fail: 2330212 0.2782464027404785 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2340843/stdout:fail: 2340843 0.22100067138671875 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2376573/stdout:fail: 2376573 0.2599215507507324 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2435591/stdout:fail: 2435591 0.2041921615600586 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2486279/stdout:fail: 2486279 0.16040253639221191 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2850036/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:2850084/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:2850132/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:797729/stdout:fail: 797729 0.7897646427154541 exception ::omero::ResourceError

[will-moore]

idr-next uk1s3 steps

idr0011, idr0013, idr0016

• Grab all the latest sql scripts, now at https://github.com/IDR/mkngff_upgrade_scripts

[wmoore@prod120-omeroreadwrite ~]$ cd /uod/idr/metadata/
[wmoore@prod120-omeroreadwrite metadata]$ sudo -Es git clone https://github.com/IDR/mkngff_upgrade_scripts.git
[wmoore@prod120-omeroreadwrite mkngff_upgrade_scripts]$ git log
commit 551f04a4abfeed073e15dcf02c3f9c7e301f5adc
Author: William Moore <w.moore@dundee.ac.uk>
Date:   Fri Jan 5 12:14:44 2024 +0000

    Add idr0011_four.csv idr0013_three.csv idr0016_seven.csv
...

Update sql with SECRET...

cd idr0011  # idr0013, idr0016
for i in $(ls); do sudo sed -i 's/SECRETUUID/e3e1ac30-7b69-473b-98b2-428780578b1c/g' $i; done

Mount buckets with goofys:

sudo mkdir /idr0011 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0011 /idr0011
sudo mkdir /idr0013 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0013 /idr0013
sudo mkdir /idr0016 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0016 /idr0016

See steps at https://github.com/IDR/idr-metadata/issues/642#issuecomment-1877008467 As omero-server...

cd /uod/idr/metadata/mkngff_upgrade_scripts/ngff_filesets

omero login
export IDRID=idr0011
for r in $(cat idr0011_four.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions
done

Images from all 4 Plates are viewable 👍

idr0013....

export IDRID=idr0013
for r in $(cat idr0013_three.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions
done

Ooops - copy/paste error - still had idr0011 in symlink, and symlink doesn't have /zarr for idr0013... Fix for all 3:

rm /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr
ln -s /idr0013/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr

rm /data/OMERO/ManagedRepository/demo_2/2016-05/07/02-36-52.924_mkngff/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr
ln -s /idr0013/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/07/02-36-52.924_mkngff/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr

rm /data/OMERO/ManagedRepository/demo_2/2016-05/03/23-33-31.705_mkngff/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr
ln -s /idr0013/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/03/23-33-31.705_mkngff/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr

Images from all 3 Plates are viewable 👍

idr0016...

for r in $(cat idr0016_seven.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317451
(1 row)

COMMIT
Previous session expired for demo on localhost:4064
Server: [localhost:4064]
Username: [demo]
Password:
Created session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr -> /idr0016/zarr/25585.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317452
(1 row)
...

Images from all 7 Plates are viewable 👍

idr0064

See https://github.com/IDR/idr-metadata/issues/682#issuecomment-1860689561

sudo mkdir /idr0064 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0064 /idr0064

export IDRID=idr0064
for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0064/$platename.ome.zarr" --bfoptions
done

UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317458
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff/1.1.ome.zarr -> /idr0064/1.1.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff/1.1.ome.zarr.bfoptions
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317459
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff/1.2.ome.zarr -> /idr0064/1.2.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff/1.2.ome.zarr.bfoptions
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317460
(1 row)
...

Images from all 9 Plates are visible in webclient 👍

idr0091

See https://github.com/IDR/idr-metadata/issues/650#issuecomment-1878458714

sudo mkdir /idr0091 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0091 /idr0091

for r in $(cat idr0091_imageids.csv); do
  python /uod/idr/metadata/idr-utils/scripts/managed_repo_symlinks.py Image:$r /idr0091/zarr/ --repo /data/OMERO/ManagedRepository --fileset-mappings idr0091_symlinks.csv --report
done

...
Fileset: 6314117 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2020-10/02/15-02-19.393_mkngff/
Render Image 10648050
fs_contents ['0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr', '0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr.bfoptions']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2020-10/02/15-02-19.393_mkngff/0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr to /idr0091/zarr/20151204_switch6h_pos0_GL06.ome.zarr
Symlink target not found: /idr0091/zarr/0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr.bfoptions
...
Fileset: 6314274 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2020-10/03/18-48-59.765_mkngff/
Render Image 10648771
fs_contents ['882f80fa-f40f-455b-b923-09dce086675b.zarr', '882f80fa-f40f-455b-b923-09dce086675b.zarr.bfoptions']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2020-10/03/18-48-59.765_mkngff/882f80fa-f40f-455b-b923-09dce086675b.zarr to /idr0091/zarr/20161207_Pos1_GL26.ome.zarr
Symlink target not found: /idr0091/zarr/882f80fa-f40f-455b-b923-09dce086675b.zarr.bfoptions

[will-moore]

For idr0009 - need to add scanr.skip_missing_wells=false to a .bfoptions file for every Plate (964 plates)! E.g. ?show=well-607352 plate has missing Wells. Trying to view any image after missing Wells shows wrong Image or error.

Fileset template prefixes - test with just a couple from that Plate and the next

http://localhost:1080/webclient/?show=plate-2703 http://localhost:1080/webclient/?show=plate-2705

bash-4.2$ cat idr0009_filesetprefix.csv
demo_2/2015-11/26/07-41-09.642
demo_2/2015-11/26/07-41-14.425

For each, create experiment_descriptor.xml.bfoptions

 for f in $(cat idr0009_filesetprefix.csv); do echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$f/experiment_descriptor.xml.bfoptions"; done

This has created .bfoptions file... for http://localhost:1080/webclient/?show=plate-2705

bash-4.2$ ls -alh /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 112K
drwxrwxr-x.  3 omero-server omero-server  153 Jan  8 15:43 .
drwxrwxr-x. 72 omero-server omero-server 8.0K Nov 26  2015 ..
lrwxrwxrwx.  1 omero-server omero-server   97 Nov 26  2015 AcquisitionLog.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/AcquisitionLog.dat
drwxrwxr-x.  2 omero-server omero-server  72K Nov 26  2015 data
lrwxrwxrwx.  1 omero-server omero-server  104 Nov 26  2015 experiment_descriptor.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.dat
lrwxrwxrwx.  1 omero-server omero-server  104 Nov 26  2015 experiment_descriptor.xml -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.xml
-rw-r--r--.  1 omero-server omero-server   31 Jan  8 15:43 experiment_descriptor.xml.bfoptions

The bfmemo exists for some Plates, e.g. first Plate of idr0009 on idr-testing:

ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-22-16.896
total 448K
drwxrwxr-x. 2 omero-server omero-server   47 Nov 30 17:36 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 448K Nov 30 17:36 .experiment_descriptor.xml.bfmemo

and for Plate above http://localhost:1080/webclient/?show=plate-2705 with some missing Wells

bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server   47 Jan  8 15:37 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan  8 15:37 .experiment_descriptor.xml.bfmemo

but not for other plates with missing Wells - e.g. first plate above

bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642
total 0
drwxrwxr-x. 2 omero-server omero-server  6 Jan  8 15:58 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan  8 15:58 ..

lets try to delete .bfmemo and see if it fixes problem...

bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/.experiment_descriptor.xml.bfmemo 

Now, we can no-longer view ANY images on http://localhost:1080/webclient/?show=plate-2705

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml

👎

After I delete the bfoptions file created above

$ rm /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml.bfoptions 

This allows images to be viewable again and a new .bfmemo is created

$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server   47 Jan  8 16:41 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan  8 16:41 .experiment_descriptor.xml.bfmemo

[will-moore]

idr0009 .bfoptions creation

See https://github.com/IDR/idr-metadata/issues/684

On idr-next (prod120) as omero-server...

Created idr0009_filesets.csv

9509,demo_2/2015-09/23/10-40-23.554/
12601,demo_2/2015-09/30/09-20-50.608/
9512,demo_2/2015-09/23/10-48-58.762/
...
16663,demo_2/2015-11/26/09-12-37.788/
16664,demo_2/2015-11/26/09-15-28.047/
16665,demo_2/2015-11/26/09-15-36.785/

Also created addbfoptions.sql

CREATE OR REPLACE FUNCTION add_origfile_to_fileset(
    fileset_id bigint,
    name character varying,
    uuid character varying)
  RETURNS integer AS
$BODY$
DECLARE
   new_event integer;
   old_owner integer;
   old_group integer;
   old_perms integer;
   new_file integer;
   files_count integer;
   prefix character varying;

BEGIN

    select _current_or_new_event() into new_event;
    select
        owner_id, group_id, permissions, templateprefix
     into
        old_owner, old_group, old_perms, prefix
     from fileset where id = fileset_id;

    select count(id) from filesetentry where fileset = fileset_id into files_count;

    insert into originalfile (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
        values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            'application/octet-stream', 'cdf35825-def1-4580-8d0b-9c349b8f78d6', prefix, uuid || name)

            returning id into new_file;

    insert into filesetentry
          (id, permissions, creation_id, group_id, owner_id, update_id, fileset, originalfile, fileset_index, clientpath)
          values (nextval('seq_filesetentry'), old_perms, new_event, old_group, old_owner, new_event,
                  fileset_id, new_file, files_count, name);

    RETURN new_file;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE;

$ cat idr0009_filesets.csv | wc
   1034    1034   39125

$ psql -U omero -d idr -h $DBHOST -f addbfoptions.sql 
CREATE FUNCTION

$ for r in $(cat idr0009_filesets.csv); do
  fsid=$(echo $r | cut -d',' -f1)
  prefix=$(echo $r | cut -d',' -f2 | sed 's/.$//')
  echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$prefix/experiment_descriptor.xml.bfoptions";
  psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'experiment_descriptor.xml.bfoptions', '$SECRET')"
  rm -f /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/.experiment_descriptor.xml.bfmemo 
done

[will-moore]

Snapshot of idr-next just taken by Dom.

Upate idr-next to "revert" usage of our own s3 buckets at https://github.com/IDR/idr-metadata/issues/675#issuecomment-1878828866 and replace the NGFF Filesets created there with ones based on BioStudies-hosted data (but without adding an extra _mkngff suffix to the Fileset.templatePrefix....

Needs install of https://github.com/IDR/omero-mkngff/pull/14 to create new Filesets without extra _mkngff suffix... And use --fs_suffix=None below...

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@fs_suffix'

idr0011.csv - with Fileset IDs from idr-next...

idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/684211eb-77bc-4693-a904-851c79f19b28,6317438
idr0011/Plate13-Red-B.ome.zarr,S-BIAD866/a3781c5d-c4e2-4ad6-9d57-357069721845,6317439
idr0011/Plate3-Yellow-B.ome.zarr,S-BIAD866/311935f7-f3d5-4405-9456-d8c49f736f11,6317440
idr0011/Plate11-Blue-A.ome.zarr,S-BIAD866/e138116a-2215-4581-928e-73c32d667018,6317441
idr0011/Target-TS,S-BIAD866/ef7df8ac-0c98-47a6-944e-6e86f18f174a,6312743

idr0013.csv - with Fileset IDs from idr-next...

idr0013/LT0080_37.ome.zarr.zip,S-BIAD865/aea4aa32-60c2-4a38-8a91-9f303381e562,6317449
idr0013/LT0066_23.ome.zarr.zip,S-BIAD865/c1d9f06e-cfd0-43cd-be2f-3e5f39c3b62a,6317448
idr0013/LT0103_13.ome.zarr.zip,S-BIAD865/eae9bb4c-9504-4f88-9931-dbf234f86023,6317450

idr0016.csv - with Fileset IDs from idr-next...

idr0016/25585.ome.zarr,S-BIAD851/da7853ca-2f99-4b46-b159-00371ffbe4c3,6317451
idr0016/24634.ome.zarr,S-BIAD851/cd887d08-ea47-489f-ab2d-039a64af938b,6317452
idr0016/24667.ome.zarr,S-BIAD851/adbbfaf5-5ac4-48f1-acf7-3180b3a6445d,6317453
idr0016/25909.ome.zarr,S-BIAD851/076d5d53-e59f-40cd-bae9-c914c7a1ae26,6317454
idr0016/25592.ome.zarr,S-BIAD851/539f798f-99c9-4351-a4ca-35c6ad4f2d22,6317455
idr0016/25675.ome.zarr,S-BIAD851/312f9dbd-eaec-4993-b0c5-95c9db5a0248,6317456
idr0016/25740.ome.zarr,S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631,6317457

For each of idr0011, idr0013, idr0016...

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --fs_suffix=None --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

for i in $(ls); do sed -i 's/SECRETUUID/e3e1ac30-7b69-473b-98b2-428780578b1c/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --fs_suffix=None --bfoptions
done

...
# idr0013
UPDATE 368
BEGIN
 mkngff_fileset 
----------------
        6317474
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr -> /bia-integrator-data/S-BIAD865/eae9bb4c-9504-4f88-9931-dbf234f86023/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr.bfoptions

#idr0016
...
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317481
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr -> /bia-integrator-data/S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631/759823ab-291d-460b-bfca-9f8903e50631.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr.bfoptions

idr0011:

• http://localhost:1080/webclient/?show=plate-5639

[will-moore]

Same for idr0064...

idr0064.csv - IDs for idr-next...

idr0064/1.1.ome.zarr,S-BIAD992/ec0b496e-2d48-44ed-be4d-0339f8927eef,6317458
idr0064/1.2.ome.zarr,S-BIAD992/352f2cac-020d-494d-8e48-37a8f4e4f3f2,6317459
idr0064/1.3.ome.zarr,S-BIAD992/34948485-1bde-4da6-a4ca-c21e223092fc,6317460
idr0064/2.1.ome.zarr,S-BIAD992/641125ba-978c-488f-be0f-75b9bb09b6f6,6317461
idr0064/2.2.ome.zarr,S-BIAD992/4518e5a4-7804-4918-b42a-678ebd89ba0d,6317462
idr0064/2.3.ome.zarr,S-BIAD992/e40d09f6-152a-47c1-8986-a5f924d608f8,6317463
idr0064/3.1.ome.zarr,S-BIAD992/23d753d0-de0b-4f72-8b99-1fe39605cfc0,6317464
idr0064/3.2.ome.zarr,S-BIAD992/bf6dfdb8-d84c-47f6-a36a-11d1dddfe689,6317465
idr0064/3.3.ome.zarr,S-BIAD992/16d9a6cb-32d3-49a3-8654-9c869baa2394,6317466

...
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317490
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr -> /bia-integrator-data/S-BIAD992/16d9a6cb-32d3-49a3-8654-9c869baa2394/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr.bfoptions

Looks good... http://localhost:1080/webclient/?show=screen-2351

[will-moore]

For testing, all these are now hosted by BioStudies...

idr0011/Plate15-Yellow-A idr0011/Plate13-Red-B idr0011/Plate3-Yellow-B idr0011/Plate11-Blue-A idr0011/Target-TS

idr0013/LT0080_37 idr0013/LT0066_23 idr0013/LT0103_13

idr0016/25585 idr0016/24634 idr0016/24667 idr0016/25909 idr0016/25592 idr0016/25675 idr0016/25740

idr0064 (all plates)

[will-moore]

Installed geesefs and switched goofys for geesefs mounts on each of the 5 nodes: e.g.

[wmoore@prod120-omeroreadonly-4 ~]$ wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64
2024-01-24 10:00:42 (297 MB/s) - ‘geesefs-linux-amd64’ saved [31464377/31464377]

[wmoore@prod120-omeroreadonly-4 ~]$ chmod +x geesefs-linux-amd64
[wmoore@prod120-omeroreadonly-4 ~]$ mv geesefs-linux-amd64 geesefs
[wmoore@prod120-omeroreadonly-4 ~]$ sudo cp geesefs /usr/local/bin
[wmoore@prod120-omeroreadonly-4 ~]$ sudo umount /bia-integrator-data
[wmoore@prod120-omeroreadonly-4 ~]$ sudo /usr/local/bin/geesefs --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data
s3.INFO anonymous bucket detected
main.INFO File system has been successfully mounted.

[will-moore]

After testing by @pwalczysko, now switching back to use goofys on idr-next...

On each server, unmount (geesefs), re-mount (goofys), restart server...

[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo umount /bia-integrator-data"; done
[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data"; done
[wmoore@prod120-proxy ~]$ 
[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service

Message from syslogd@localhost at Jan 24 12:12:21 ...
 haproxy[19461]: backend omero4064-0 has no server available!
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service

Message from syslogd@localhost at Jan 24 12:13:20 ...
 haproxy[19461]: backend omero4064-1 has no server available!

[will-moore]

start from scratch on rocky9 pilot

• Update ZarrReader to today's build, including https://github.com/ome/ZarrReader/pull/82

$ ssh pilot-idrngff -L 1080:localhost:80
$ sudo -u omero-server -s
$ cd
$ wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/33/label=testintegration/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
bash-5.1$ mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_b33.jar
bash-5.1$ rm OMERO.server/lib/server/OMEZarrReader.jar 
bash-5.1$ rm OMERO.server/lib/client/OMEZarrReader.jar 
bash-5.1$ cp OMEZarrReader_b33.jar OMERO.server/lib/server/
bash-5.1$ cp OMEZarrReader_b33.jar OMERO.server/lib/client/

[wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo service omero-server restart

• Install mkngff in venv, including recent PR branches (if not yet merged) - To include https://github.com/IDR/omero-mkngff/pull/17

$ sudo -u root -s
# source /opt/omero/server/venv3/bin/activate

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@alt_store_bfoptions'

• As omero-server, clone https://github.com/IDR/mkngff_upgrade_scripts

$ sudo -u omero-server -s
bash-5.1$ cd
bash-5.1$ git clone https://github.com/IDR/mkngff_upgrade_scripts.git

• Delete duplicate plate for idr0015

screen -S mkngff
...
$ omero delete Plate:4801 --report > /tmp/delete_idr0015.log

• Get $SECRET, update all sql commands with it and run them (see https://github.com/IDR/mkngff_upgrade_scripts)

for i in $(ls); do sed -i 's/SECRETUUID/e1e69d03-6642-4f0b-8c0c-c98c1f89fb4b/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr"
done

• idr0004 - remove zarrreader.quick_read=true from bfoptions for 2 plates...

Plate P115:

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions

Plate P124:

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-57-40.271_mkngff/ee8872c8-e4b1-41fa-aa4f-a9e3e200c540.zarr.bfoptions

Try to view image from 2nd Plate at 9:40...http://localhost:1080/webclient/?show=well-469196 - success ~ 10:00. Also Plate P115 was viewable.

• idr0010 - sql applied - viewing first plate 12:12 (15th Aug)...
• idr0011 - sql applied... (start 12:16 15th Aug)...
• idr0012 - sql applied... (start 13:15)
• idr0013 - sql applied, start 13:42...
• idr0015 - sql applied, start 15:13....
• idr0016 - sql applied, start 16:21...
• idr0025 - sql applied, start 19:25...
• idr0026 - sql applied, start 20:00...
• idr0035 - sql applied, start 20:35...
• idr0036 - sql applied, start 22:29...
• idr0051 - sql applied, start 22:42...
• idr0054 - sql applied, start 22:45...
• idr0064 - sql applied, start 22:53...
• idr0090 - sql applied, start 22:57...
• idr0091 - sql applied, start 23:04...

[will-moore]

Since idr0091 scripts (idr0091.csv) was not updated back to the original Fileset IDs, the wrong Filesets got replaced on pilot-idrngff above. Now, with the csv updated with correct IDs, we can't run this on pilot-idrngff because the OriginalFile path/names we want to create are already there and we aren't allowed duplicates! This could take a long time to clean-up, so an easier solution is to test idr0091 on a different server...

Use pilot-rocky9:

$ ssh pilot-rocky9

• Added ZarrReader (build 41) as above (no OMEZarrReader.jar there before) but didn't restart server yet... (Dom using it).
• installed omero-mkngff
• Cloned mkngff_upgrade_scripts
• Ran omero mkngff sql to regenerate idr0091 sql scripts with original IDs. Committed at https://github.com/IDR/mkngff_upgrade_scripts/commit/500c9970dd030c10c6d1f583b1764da0cd64c13b
• Then...

cd idr0091
for i in $(ls); do sed -i 's/SECRETUUID/bc9a125d-fa7c-417c-a33b-506390bb3b7f/g' $i; done
cd ../
omero login
...
# applied sql scripts, symlinks, bfoptions etc.

BUT - this data isn't viewable since we don't have latest Bio-Formats on this server. Just the IDR version.

So, back to the drawing board...!

EDIT:

• deleted all the incorrectly generated filesets above
• ran idr0091 scripts from github
• all OK 👍

[will-moore]

idr0009 update on pilot-idrngff as recorded at https://github.com/IDR/idr-metadata/issues/684#issuecomment-2076629628

[will-moore]

Testing of svs update failed due to Bio-Formats version mismatches...

David: "You should be able to replace the jars, the main jars are formats-api, formats-bsd, formats-gpl and turbojpeg. If that has the latest snapshot then you might also need to check the lower level components, so anything listed under component updates in https://www.openmicroscopy.org/2024/04/18/bio-formats-7-3-0.html has since been updated"

Downloaded various jars into bfjars_25thApril_b48:

bash-5.1$ ls -alh bfjars_25thApril_b48/
total 7.2M
drwxr-xr-x.  2 omero-server omero-server 4.0K Apr 25 12:43 .
drwxr-xr-x. 11 omero-server root         4.0K Apr 25 12:46 ..
-rw-r--r--.  1 omero-server omero-server 143K Apr 25 00:08 formats-api-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 880K Apr 25 00:08 formats-bsd-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.5M Apr 25 00:08 formats-gpl-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  14K Apr 25 00:08 metakit-5.3.8-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 7.8K Apr 25 00:08 mipav-stubs-6.0.3-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  90K Apr 25 00:08 ome-codecs-1.0.4-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 139K Apr 25 00:07 ome-common-6.0.23-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.0M Apr 25 00:08 ome-jai-0.1.5-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  90K Apr 25 00:08 ome-mdbtools-5.3.4-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 928K Apr 25 00:07 ome-poi-5.3.10-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 639K Apr 25 00:07 ome-xml-6.3.7-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 615K Apr 25 00:07 specification-6.3.7-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.3M Apr 25 00:08 turbojpeg-8.0.0-SNAPSHOT.jar

For each of these, moved the existing equivalent jar from OMERO.server/lib/server into bfjars_orig, and deleted jar from OMERO.server/lib/client. NB: couldn't find existing jar equivalent to mipav-stubs-6.0.3-SNAPSHOT.jar.

bash-5.1$ ls -alh bfjars_orig
total 7.2M
drwxr-xr-x.  2 omero-server omero-server 4.0K Apr 25 12:56 .
drwxr-xr-x. 11 omero-server root         4.0K Apr 25 12:46 ..
-rw-r--r--.  1 omero-server omero-server 140K Apr 11 14:02 formats-api.jar
-rw-r--r--.  1 omero-server omero-server 877K Apr 11 14:02 formats-bsd.jar
-rw-r--r--.  1 omero-server omero-server 1.5M Apr 22 12:15 formats-gpl-8.0.0-SNAPSHOT_b45.jar
-rw-r--r--.  1 omero-server omero-server  14K Apr 11 14:02 metakit.jar
-rw-r--r--.  1 omero-server omero-server  89K Apr 11 14:02 ome-codecs.jar
-rw-r--r--.  1 omero-server omero-server 140K Apr 11 14:02 ome-common.jar
-rw-r--r--.  1 omero-server omero-server 1.1M Apr 11 14:02 ome-jai.jar
-rw-r--r--.  1 omero-server omero-server  91K Apr 11 14:02 ome-mdbtools.jar
-rw-r--r--.  1 omero-server omero-server 930K Apr 11 14:02 ome-poi.jar
-rw-r--r--.  1 omero-server omero-server 638K Apr 11 14:02 ome-xml.jar
-rw-r--r--.  1 omero-server omero-server 615K Apr 11 14:02 specification.jar
-rw-r--r--.  1 omero-server omero-server 1.3M Apr 11 14:02 turbojpeg.jar

Then copied all new jars into OMERO.server/lib/server/ and OMERO.server/lib/client/.

cp bfjars_25thApril_b48/* OMERO.server/lib/server/
cp bfjars_25thApril_b48/* OMERO.server/lib/client/

Restarted server...