idr0016-wawer-bioactivecompoundprofiling S-BIAD851

[dominikl]

https://github.com/IDR/idr0016-wawer-bioactivecompoundprofiling

Sample plate conversion failed with:

(base) [dlindner@pilot-zarr2-dev idr0016]$ time /home/dlindner/bioformats2raw/bin/bioformats2raw --memo-directory ../memo 24320.screen 24320.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp6586654250319720590/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
[Fatal Error] :1:84: Character reference "&#0" is an invalid XML character.
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@63a65a25): java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 84; Character reference "&#0" is an invalid XML character.
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)

This error Character reference "&#0" is an invalid XML character is already referenced by https://github.com/IDR/bioformats/issues/29 .

[will-moore]

checking error above for http://localhost:1040/webclient/?show=image-2376573 Fileset ID: 5288909 (new) Plate name: 25675

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/.zattrs

Other Images on the same plate have same path/name

idr=> select path, name from pixels where image=2376573;
                                        path                                         |  name   
-------------------------------------------------------------------------------------+---------
 demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1 | .zattrs
(1 row)

idr=> select path, name from pixels where image=2378563;
                                        path                                         |  name   
-------------------------------------------------------------------------------------+---------
 demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1 | .zattrs
(1 row)

idr0016.csv row:

idr0016/25675.ome.zarr,S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe,21385

21385.sql is missing ...zarr/.zattrs and all A/1 etc:

UPDATE pixels SET name = '.zattrs', path = 'demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1' where image in (select id from Image where fileset = 21385);

begin;
    select mkngff_fileset(
      21385,
      'c6b02bb7-2c22-4c45-be8d-30484c380a9c',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/2016-06/21/01-46-55.560_mkngff/',
      array[
          ['demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/', '.zgroup', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/.zgroup'],
          ['demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/', '.zattrs', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/.zattrs'],

Invalid: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr

Check for other plates that have fewer files than expected:...

$ grep -c https ./*.sql
./21206.sql:14610
./21207.sql:14610
./21208.sql:14610
./21209.sql:14610

./21272.sql:5906
./21334.sql:14572
./21363.sql:14607
./21365.sql:14602
./21369.sql:14603
./21385.sql:13695
./21451.sql:1802
./21476.sql:14602
./21548.sql:14607
./21551.sql:14154

./21552.sql:14420
./21555.sql:14344
./21558.sql:574
./21559.sql:2672
./21560.sql:2101
./21561.sql:3319
./21562.sql:3242
./21563.sql:3965
./21564.sql:8643
./21565.sql:10544
./21566.sql:12672
./21567.sql:14268
./21569.sql:14268
./21570.sql:14534
./21571.sql:13774
./21572.sql:13926
./21573.sql:14306
./21574.sql:14572

./21582.sql:14344
./23603.sql:14603

[will-moore]

Checking http://localhost:1040/webclient/?show=image-2330212 on idr0125-pilot... Finally gave a webclient error:

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs

$ grep -A 5 -B 5 "995c-83b7cbf5372f" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
...
2023-12-05 14:02:34,787 INFO  [ome.services.sessions.state.SessionCache] (2-thread-5) Synchronization took 71 ms.
2023-12-05 14:02:36,183 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1701784193425] time[762757] tag[loci.formats.Memoizer.setId]
2023-12-05 14:02:36,185 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-1) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs
2023-12-05 14:02:36,187 ERROR [                ome.io.nio.PixelsService] (l.Server-1) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs
java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
    at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
    at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)

Checking ALL Wells at https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fbia-integrator-data%2FS-BIAD851%2F1b63ff26-391a-4b44-995c-83b7cbf5372f%2F1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr&well=all Finds 2 Images invalid: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/4/ missing...zarray Error Loading https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/4/2/.zarray: Not Found

Missing .zattrs at

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/5/

https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/5/.zattrs: Not Found```

[will-moore]

http://localhost:1040/webclient/?show=image-2340843

idr0016.csv

idr0016/25592.ome.zarr,S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5,21369

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 5 -B 5 "fa2c106c7cf5" /opt/omero/server/OMERO.server/var/log/Blitz-0.log

--
2023-12-05 12:50:45,581 INFO  [         ome.security.basic.EventHandler] (l.Server-1)  Auth:    user=52,group=3,event=null(User),sess=3e1880d1-c38b-46b1-8e4c-460886f220fb
2023-12-05 12:50:45,586 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1701780645579] time[6] tag[omero.call.success.ome.logic.AdminImpl.lookupGroups]
2023-12-05 12:50:45,586 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Rslt:    (ome.model.meta.ExperimenterGroup:Id_0, ome.model.meta.ExperimenterGroup:Id_1, ome.model.meta.ExperimenterGroup:Id_2, ... 3 more)
2023-12-05 12:50:55,948 DEBUG [                   loci.formats.Memoizer] (l.Server-6) start[1701780110386] time[545562] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,949 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1701780107265] time[548684] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-6) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1701779997653] time[658297] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1701780089348] time[566601] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-7) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,951 ERROR [                ome.io.nio.PixelsService] (l.Server-6) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
    at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
    at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
--

1 Well from the whole plate has errors: 2 Files missing at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/J/12/4/

1 at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/J/12/5/

[will-moore]

http://localhost:1040/webclient/?show=image-2486279

idr0016.csv

idr0016/25909.ome.zarr,S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015,21476

Found 1 Well with errors:

Missing 3 files from https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015/382abcd4-404b-4796-bbe2-b02fedcbb015.zarr/A/5/2/

and at least 1 from https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015/382abcd4-404b-4796-bbe2-b02fedcbb015.zarr/A/5/3/

[will-moore]

http://localhost:1040/webclient/?show=image-2131185

idr0016/24634.ome.zarr,S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84,21278

One Well invalid: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/

2 missing files: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/4/

1 missing .zattrs https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/5/

[will-moore]

http://localhost:1040/webclient/?show=image-2435591

idr0016/25740.ome.zarr,S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e,21454

1 Well invalid:

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/

3 Files missing: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/3/

.zattrs missing https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/4/

[will-moore]

Need to re-export broken plates...

Plate name: 25585 ID: 5065 Plate name: 25592 ID: 5069 Plate name: 25675 ID: 5085 Plate name: 25909 ID: 5176 Plate name: 24634 ID: 4978 Plate name: 25740 ID: 5154

for id in 5065 5069 5085 5176 4978 5154; do
  echo $id;
  omero zarr export Plate:$id --name_by name;
done

EDIT: 6th Dec. zipping 6 plates...

for i in */; do zip -r "${i%/}.zip" "$i"; done

[will-moore]

(base) [wmoore@idrftp-ftp idr0004]$ sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d /data/idr0016/idr0016/ bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/1xxxxxxxxxx      
24634.ome.zarr.zip                                                                      100% 5683MB 94.0Mb/s    06:44
25585.ome.zarr.zip                                                                      100% 5986MB  162Mb/s    13:45
25592.ome.zarr.zip                                                                      100% 6090MB  162Mb/s    20:41
25675.ome.zarr.zip                                                                       100% 5759MB 67.7Mb/s    27:13
25740.ome.zarr.zip                                                                       100% 5695MB  147Mb/s    33:29    
25909.ome.zarr.zip                                                                      100% 5949MB  244Mb/s    38:58    
Completed: 36009422K bytes transferred in 2338 seconds
 (126141K bits/sec), in 6 files, 1 directory.

[will-moore]

Checking https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD851.html Comparing with rows removed from idr0016.csv at https://github.com/IDR/idr-utils/pull/56/commits/125c4e559d5020dacbf359bc2bf7694dd86fb708

Unchanged: idr0016/24634.ome.zarr K19 still missing: https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fbia-integrator-data%2FS-BIAD851%2F22803be2-9732-41ab-b9bf-fcd37c3b3b84%2F22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr&well=all

idr0016/25585 https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr

...

[will-moore]

Let's upload 7 plates to uk1s3 for testing and mkngff... First command timed-out on last 2 plates - re-ran those 2...

(base) [wmoore@idrftp-ftp ~]$ ls /data/idr0016/idr0016
24634.ome.zarr  24667.ome.zarr  25585.ome.zarr  25592.ome.zarr  25675.ome.zarr  25740.ome.zarr  25909.ome.zarr

./mc cp -r /data/idr0016/idr0016/ uk1s3/idr0016/zarr
.../idr0016/25675.ome.zarr/K/3/5/2/2/0/0: 29.50 GiB / 30.50 GiB ━━━━━━━━━━━━ 18.09 MiB/s

(base) [wmoore@idrftp-ftp ~]$ ./mc cp -r /data/idr0016/idr0016/25740.ome.zarr/ uk1s3/idr0016/zarr/25740.ome.zarr
...6/25740.ome.zarr/P/9/5/3/4/0/0: 6.27 GiB / 6.27 GiB ━━━━━━━━━━━━━━━━ 19.02 MiB/s 5m37s

(base) [wmoore@idrftp-ftp ~]$ ./mc cp -r /data/idr0016/idr0016/25909.ome.zarr/ uk1s3/idr0016/zarr/25909.ome.zarr
...6/25909.ome.zarr/P/9/5/3/4/0/0: 6.42 GiB / 6.42 GiB ━━━━━━━━━━━━━ 16.89 MiB/s 6m29s

on idr0125-pilot..

sudo mkdir /idr0016 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0016 /idr0016

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls /idr0016/zarr/
24279.ome.zarr  24634.ome.zarr  24667.ome.zarr  25585.ome.zarr  25592.ome.zarr  25675.ome.zarr  25740.ome.zarr

Validated all Wells in the 7 Plates, e.g. https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fidr0016%2Fzarr%2F25909.ome.zarr&well=all

[will-moore]

idr0016.csv based on https://github.com/IDR/idr-utils/commit/125c4e559d5020dacbf359bc2bf7694dd86fb708

idr0016/25585.ome.zarr,25585.ome.zarr,21365
idr0016/24634.ome.zarr,24634.ome.zarr,21278
idr0016/24667.ome.zarr,24667.ome.zarr,23603
idr0016/25909.ome.zarr,25909.ome.zarr,21476
idr0016/25592.ome.zarr,25592.ome.zarr,21369
idr0016/25675.ome.zarr,25675.ome.zarr,21385
idr0016/25740.ome.zarr,25740.ome.zarr,21454

On idr0125-pilot, logged-in to idr.openmicroscopy.org...

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0016/zarr/$platename" "/idr0016/zarr/$platename" > "$IDRID/$fsid.sql"
done

(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ for i in $(ls idr0016); do grep -c ".zattrs" idr0016/$i; done
2690
2690
2690
2690
2690
2690
2690

for i in $(ls); do sed -i 's/SECRETUUID/9630ba1e-ed3a-42e3-9296-59ccf23a7039/g' $i; done

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

Updated in https://github.com/IDR/mkngff_upgrade_scripts/commit/0ba84be770aae8ea0a9bd4961fa9579fc144f278

We can't test mkngff on idr0125-pilot as Filesets have already been updated and IDs no longer valid. Use idr0138-pilot instead...

[will-moore]

On idr0138-pilot...

As root, update...

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@bfoptions'

Get sql and update...

cd /uod/idr/metadata
sudo -Es git clone https://github.com/IDR/mkngff_upgrade_scripts.git
cd mkngff_upgrade_scripts/ngff_filesets/idr0016
for i in $(ls); do sudo sed -i 's/SECRETUUID/de86d158-d945-4256-a8c0-de34d689ca0c/g' $i; done

# csv content from above...
sudo vi idr0016_7.csv

As omero-server...

omero login
omero mkngff setup > setup.sql
psql -U omero -d idr -h $DBHOST -f setup.sql

cd /uod/idr/metadata/mkngff_upgrade_scripts/ngff_filesets/
for r in $(cat idr0016_7.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        5815205
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr -> /idr0016/zarr/25585.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr.bfoptions
UPDATE 2304
BEGIN
...

[will-moore]

On idr-testing... idr0016.csv:

idr0016/25585.ome.zarr,S-BIAD851/da7853ca-2f99-4b46-b159-00371ffbe4c3,6313495
idr0016/24634.ome.zarr,S-BIAD851/cd887d08-ea47-489f-ab2d-039a64af938b,6313510
idr0016/24667.ome.zarr,S-BIAD851/adbbfaf5-5ac4-48f1-acf7-3180b3a6445d,6313525
idr0016/25909.ome.zarr,S-BIAD851/076d5d53-e59f-40cd-bae9-c914c7a1ae26,6313540
idr0016/25592.ome.zarr,S-BIAD851/539f798f-99c9-4351-a4ca-35c6ad4f2d22,6313570
idr0016/25675.ome.zarr,S-BIAD851/312f9dbd-eaec-4993-b0c5-95c9db5a0248,6313600
idr0016/25740.ome.zarr,S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631,6313840

export IDRID=idr0016
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --fs_suffix=None --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

for i in $(ls); do sed -i 's/SECRETUUID/f464e059-16b5-4013-b9a2-417e5976371c/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --fs_suffix=None --bfoptions
done
...

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6314910
(1 row)
COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr -> /bia-integrator-data/S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631/759823ab-291d-460b-bfca-9f8903e50631.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr.bfoptions