idr0015-colin-taraoceans S-BIAD861

[will-moore]

idr0015-colin-taraoceans

[will-moore]

Failed just now on idr0125-pilot:

Found prefix: demo_2/2016-06/13/08-55-20.187 for fileset: 21180
Traceback (most recent call last):
  File "/opt/omero/server/venv3/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/main.py", line 126, in main
    rv = omero.cli.argv()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1787, in argv
    cli.invoke(args[1:])
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1225, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1302, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1384, in execute
    args.func(args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 195, in sql
    for row_path, row_name, row_mime in self.walk(symlink_path):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 295, in walk
    yield from self.walk(p)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 295, in walk
    yield from self.walk(p)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 284, in walk
    for p in path.iterdir():
  File "/usr/lib64/python3.6/pathlib.py", line 1081, in iterdir
    for name in self._accessor.listdir(self):
  File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
ConnectionAbortedError: [Errno 103] Software caused connection abort: '/bia-integrator-data/S-BIAD861/765da6ed-392a-4448-9139-5a97333b4732/765da6ed-392a-4448-9139-5a97333b4732.zarr/P/2'
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/06/14-30-00.279 for fileset: 21115
Traceback (most recent call last):
  File "/opt/omero/server/venv3/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/main.py", line 126, in main
    rv = omero.cli.argv()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1787, in argv
    cli.invoke(args[1:])
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1225, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1302, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1384, in execute
    args.func(args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 181, in sql
    if not symlink_path.exists():
  File "/usr/lib64/python3.6/pathlib.py", line 1336, in exists
    self.stat()
  File "/usr/lib64/python3.6/pathlib.py", line 1158, in stat
    return self._accessor.stat(self)
  File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
OSError: [Errno 107] Transport endpoint is not connected: '/bia-integrator-data/S-BIAD861/78ea6e5b-3c05-4731-87da-a0d2b4aed0a8/78ea6e5b-3c05-4731-87da-a0d2b4aed0a8.zarr'
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public

[will-moore]

Re-mounted goofys and restarted...

(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3);   omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql"; done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/13/08-55-20.187 for fileset: 21180

[will-moore]

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/06/00-22-29.942 for fileset: 21111
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-05/16/11-35-01.023 for fileset: 19237
Traceback (most recent call last):
  File "/opt/omero/server/venv3/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/main.py", line 126, in main
    rv = omero.cli.argv()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1787, in argv
    cli.invoke(args[1:])
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1225, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1302, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1384, in execute
    args.func(args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 195, in sql
    for row_path, row_name, row_mime in self.walk(symlink_path):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 295, in walk
    yield from self.walk(p)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 295, in walk
    yield from self.walk(p)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 295, in walk
    yield from self.walk(p)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_mkngff/__init__.py", line 284, in walk
    for p in path.iterdir():
  File "/usr/lib64/python3.6/pathlib.py", line 1081, in iterdir
    for name in self._accessor.listdir(self):
  File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
ConnectionAbortedError: [Errno 103] Software caused connection abort: '/bia-integrator-data/S-BIAD861/f96eef49-2c7c-4689-a722-d1399fb72c3a/f96eef49-2c7c-4689-a722-d1399fb72c3a.zarr/R/13/0'

Last 2: 21112 (failed before) and 19237 only...

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3)
  omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

[will-moore]

A couple of sql are mostly empty:

21112.sql
21160.sql

e.g. (NB: extra Cancelled got appended)

$ cat idr0015/21160.sql
Cancelled
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/OME' where image in (select id from Image where fileset = 21160);

begin;
    select mkngff_fileset(
      21160,
      'SECRETUUID',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/2016-06/10/13-37-45.953_mkngff/',
      array[
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/', '.zattrs', 'application/octet-stream'],
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/', '.zgroup', 'application/octet-stream'],
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/', 'OME', 'Directory'],
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/OME/', '.zattrs', 'application/octet-stream'],
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/OME/', '.zgroup', 'application/octet-stream'],
          ['demo_2/2016-06/10/13-37-45.953_mkngff/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/OME/', 'METADATA.ome.xml', 'application/octet-stream']
      ]::text[][]
    );
commit;

Strange that we can list files OK...

$ ls -alh /bia-integrator-data/S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr
total 69K
drwxr-xr-x. 2 root root 4.0K Aug 31 05:36 .
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 ..
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 A
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 B
...
$ ls -alh /bia-integrator-data/S-BIAD861/cfc34119-cbc6-4734-aa75-6923d3bd8c06/cfc34119-cbc6-4734-aa75-6923d3bd8c06.zarr
total 77K
drwxr-xr-x. 2 root root 4.0K Aug 31 18:34 .
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 ..
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 A
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 B
...

[will-moore]

For both these NGFF Plates, the top-level column dirs are missing the .zgroup files, so they get ignored.

ls -alh /bia-integrator-data/S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/A/
total 80K
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 .
drwxr-xr-x. 2 root root 4.0K Aug 31 06:02 ..
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 1
...
ls -alh /bia-integrator-data/S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/A/1
total 12K
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 .
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 ..
drwxr-xr-x. 2 root root 4.0K Sep 25 21:52 0

[will-moore]

Download a sample zip and check...

ssh idr-ftp
cd /data/ngff/idr0015
sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136e8d-e575-4755-9ac2-aa7fc10cae67-a26596/idr0015/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip ./

TARA_HCS1_H5_G100007665_G100007576--2013_10_2                                                                                                                                        100%   58GB  390Mb/s    19:33    
Completed: 61209699K bytes transferred in 1173 seconds
 (427357K bits/sec), in 1 file.

unzip...

Checking for .zgroup, there are some, but none at the /A/.zgroup level.

(base) [wmoore@idrftp-ftp idr0015]$ find ./ -name ".zgroup" | wc
    284     284   28246
(base) [wmoore@idrftp-ftp idr0015]$ find ./ -name ".zgroup"
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/OME/.zgroup
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/.zgroup
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/1/0/.zgroup
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/2/0/.zgroup
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/3/0/.zgroup
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/4/0/.zgroup
...
(base) [wmoore@idrftp-ftp idr0015]$ find ./ -name ".zarray" | wc
   1126    1126  114276
(base) [wmoore@idrftp-ftp idr0015]$ find ./ -name ".zarray"
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/1/0/0/.zarray
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/1/0/1/.zarray
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/1/0/2/.zarray
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/1/0/3/.zarray
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/2/0/0/.zarray
./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/A/2/0/1/.zarray
...

Need to regenerate! :(

[will-moore]

ssh pilot-zarr1-dev
cd /data/idr0015
screen -r idr0015_ngff

for i in TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01 TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen ${i%.*}.ome.zarr; done

TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp7672587622881757063/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/wmoore/bioformats2raw-0.6.0-24/lib/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@37271612): java.lang.NullPointerException
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2192)
Caused by: java.lang.NullPointerException
        at ome.xml.meta.OMEXMLMetadataImpl.getWellSampleImageRef(OMEXMLMetadataImpl.java:5205)
        at com.glencoesoftware.bioformats2raw.Converter.hasValidPlate(Converter.java:2055)
        at com.glencoesoftware.bioformats2raw.Converter.convert(Converter.java:604)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:516)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:107)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        ... 9 more

But the other plate converted successfully, and has all .zgroup as expected:

(omero_zarr_export) [wmoore@pilot-zarr1-dev idr0015]$ find ./TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr -name ".zgroup" | wc
    816     816   80292

[will-moore]

Tried failing plate again...

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr

Same error as above

[dgault]

Very odd if the dataset hasnt been modified at all and was successful last time. I will have to debug to see exactly what the failure is

[will-moore]

@dgault Yes - I wonder if I was in a different python environment at the time - I'll try and check - dependencies might be different...

[will-moore]

No - same error with the conda activate bioformats2raw env I have on pilot-zarr1-dev. Strange that it worked for one plate and not the other.

[will-moore]

Plate https://idr.openmicroscopy.org/webclient/?show=plate-4653 on IDR is not visible, but NGFF version looks good: https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/73d2e66a-a737-46b8-b174-6d60e9145b45/73d2e66a-a737-46b8-b174-6d60e9145b45.zarr Hopefully this will be fixed with the upgrade - NB: thumbnails will need generating!

[will-moore]

One plate failed to extract for EBI - re-converting...

ssh pilot-zarr1-dev
cd /data/idr0015
screen -r idr0015_ngff

~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.screen TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr

... looks good:

(base) [wmoore@pilot-zarr1-dev idr0015]$ find ./TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr/ -name ".zgroup" | wc
    816     816   80292

[will-moore]

Still failing to convert last plate on pilot-zarr1-dev - same error as above...

(omero_zarr_export) [wmoore@pilot-zarr1-dev idr0015]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr

Tried with latest release of bioformats2raw 0.7.0 from https://github.com/glencoesoftware/bioformats2raw/releases/tag/v0.7.0...

(base) [wmoore@pilot-zarr1-dev idr0015]$ ~/bioformats2raw-0.7.0/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr
...
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2826)
Caused by: loci.formats.UnknownFormatException: Unknown file format: /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen

But I remember we need an IDR-specific version that recognises .screen files. There is only the 1 release that we are already using: https://github.com/IDR/bioformats2raw/releases

[will-moore]

On https://www.ebi.ac.uk/biostudies/submissions/files?path=%2Fuser%2Fidr0015 Deleted the 2 filesets that we have managed to re-convert...

TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr (failed to extract)
TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr (missing .zgroup)

Zip these on pilot-zarr1-dev...

screen -S idr0015_zip
cd /data/idr0015
for i in */; do zip -mr "${i%/}.zip" "$i"; done

[dgault]

@will-moore, I was trying to reproduce the same issue you had seen in https://github.com/IDR/idr-metadata/issues/645#issuecomment-1735314753. When running the same command as you using the IDR release https://github.com/IDR/bioformats2raw/releases/tag/v0.6.0-24, it looks as though it ran successfully without the same exception. Would you be able to try again using that release linked? (the releases on the Glencoe repos wont have the IDR specific changes included)

[will-moore]

Just downloaded from that link (which is the same version I was using before), but still seeing the same issue...

(bioformats2raw) [wmoore@pilot-zarr1-dev idr0015]$ ./bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp9903770868831694569/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/data/idr0015/bioformats2raw-0.6.0-24/lib/kryo-2.24.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@16150369): java.lang.NullPointerException
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2192)
Caused by: java.lang.NullPointerException
        at ome.xml.meta.OMEXMLMetadataImpl.getWellSampleImageRef(OMEXMLMetadataImpl.java:5205)
        at com.glencoesoftware.bioformats2raw.Converter.hasValidPlate(Converter.java:2055)
        at com.glencoesoftware.bioformats2raw.Converter.convert(Converter.java:604)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:516)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:107)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        ... 9 more

[will-moore]

On idr0-slot3.openmicroscopy I created a new env etc...

conda create -n bioformats2raw python=3.9
conda activate bioformats2raw
conda install -c ome bioformats2raw

Downloaded the IDR bioformats2raw-0.6.0-24 and ran that on this last plate... Conversion started running OK but failed due to lack of disk space!

Thought that it's failing on pilot-zarr1-dev due to some dependency issue, so created a new env there too... But the problem persists, suggesting that it is failing on pilot-zarr1-dev due to an issue with the data.

[will-moore]

Was going to try conversion on pilot-zarr2-dev but on that machine I can't see any data:

Installed conda in my home idr...

conda create -n bioformats2raw python=3.9
conda activate bioformats2raw
conda install -c ome bioformats2raw
cd
wget https://github.com/IDR/bioformats2raw/releases/download/v0.6.0-24/bioformats2raw-0.6.0-24.zip
unzip bioformats2raw-0.6.0-24.zip

But...

(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015]$ ls /uod/idr/metadata/idr0015-colin-taraoceans/
(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015]$ 
(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015]$ ls /uod/idr/metadata/idr0015-UNKNOWN-taraoceans/
(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015]$ 

[will-moore]

$ ssh pilot-zarr2-dev
Last login: Mon Oct 16 11:41:47 2023 from pilot-proxy.sample.openstack.org
(base) [wmoore@pilot-zarr2-dev ~]$ df -h /data/idr0015
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        750G  229G  522G  31% /data

Cloned a copy here - (didn't know how to get the submodule to fetch)...

(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015-colin-taraoceans]$ cd /uod/idr/metadata/idr0015-colin-taraoceans/
(bioformats2raw) [wmoore@pilot-zarr2-dev idr0015-colin-taraoceans]$ sudo -Es git clone git@github.com:IDR/idr0015-colin-taraoceans.git
(bioformats2raw) [wmoore@pilot-zarr2-dev ~]$ cd /data/idr0015/idr0015
(bioformats2raw) [wmoore@pilot-zarr2-dev ~]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.screen TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp10574995368140289468/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

When done, checked for .zgroups... - looks good...

(base) [wmoore@pilot-zarr2-dev idr0015]$ find ./TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr/ -name ".zgroup" | wc
    816     816   80292

[will-moore]

Deleted TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip from https://www.ebi.ac.uk/biostudies/submissions/files?path=%2Fuser%2Fidr0015 and uploaded replacement...

[will-moore]

Since https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr is not valid because the .zattrs doesn't contain plate info..

$ curl https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350.zarr/.zattrs
{
  "bioformats2raw.layout" : 3
}

Need to check if any others have this issue...

mkdir idr0015_zattrs
for r in $(cat idr0015.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  curl https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr/.zattrs > idr0015_zattrs/$uuid.zarr.zattrs
done

cd idr0015_zattrs
for i in $(ls); do echo $i; grep "plate" $i | wc; done

Found one other plate has the same issue:

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/cfc34119-cbc6-4734-aa75-6923d3bd8c06/cfc34119-cbc6-4734-aa75-6923d3bd8c06.zarr

Those 2 plates are:

idr0015/TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr, S-BIAD861/90ef79de-8222-4b6d-aa4e-d4f1bd2f1350
idr0015/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr, S-BIAD861/cfc34119-cbc6-4734-aa75-6923d3bd8c06

Both are these are plates that we have already re-converted and submitted to replace originals. So hopefully we don't need to do anything else here - Just waiting on EBI to update them.

[will-moore]

On idr-testing, Plate TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01 http://localhost:1080/webclient/?show=plate-4653 has not fixed the broken thumbnails etc at https://idr.openmicroscopy.org/webclient/?show=plate-4653. Now we get ResourceError trying to view images.

Since NGFF looks good https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/73d2e66a-a737-46b8-b174-6d60e9145b45/73d2e66a-a737-46b8-b174-6d60e9145b45.zarr, let's try to import that to replace the plate (plate has no KV-pairs etc, so don't need to update annotations).

Let's import the METADATA.ome.xml then run mkngff to create the full plate...

as omero-server..

wget https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/73d2e66a-a737-46b8-b174-6d60e9145b45/73d2e66a-a737-46b8-b174-6d60e9145b45.zarr/OME/METADATA.ome.xml

omero import --transfer=ln_s --skip=all METADATA.ome.xml --file /tmp/idr0015_20231102.log  --errs /tmp/idr0015_20231102.err

2023-11-02 15:49:27,716 303        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2023-11-02 15:49:27,739 326        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.0.0 revision: 3f8b3326cb578d59bd948fb84c838ff77e9f1b08 date: 1 August 2023
2023-11-02 15:49:27,806 393        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-11-02 15:49:27,808 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-11-02 15:49:27,808 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-11-02 15:49:27,808 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-11-02 15:49:27,808 395        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-11-02 15:49:27,811 398        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-11-02 15:49:28,202 789        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2023-11-02 15:49:28,357 944        [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /opt/omero/server/idr0015/METADATA.ome.xml
loci.formats.FormatException: loci.common.services.ServiceException: Could not get OME-XML version
        at loci.formats.in.OMEXMLReader.initFile(OMEXMLReader.java:267)
        at loci.formats.FormatReader.setId(FormatReader.java:1466)
        at loci.formats.ImageReader.setId(ImageReader.java:863)
        at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at loci.formats.Memoizer.setId(Memoizer.java:698)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
        at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
        at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
        at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
Caused by: loci.common.services.ServiceException: Could not get OME-XML version
        at loci.formats.services.OMEXMLServiceImpl.transformToLatestVersion(OMEXMLServiceImpl.java:204)
        at loci.formats.services.OMEXMLServiceImpl.createOMEXMLMetadata(OMEXMLServiceImpl.java:376)
        at loci.formats.services.OMEXMLServiceImpl.createOMEXMLMetadata(OMEXMLServiceImpl.java:363)
        at loci.formats.in.OMEXMLReader.initFile(OMEXMLReader.java:261)
        ... 17 common frames omitted
2023-11-02 15:49:28,358 945        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 149ms. (156ms total) [0 unknowns]
2023-11-02 15:49:28,407 994        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-11-02 15:49:28,972 1559       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-11-02 15:49:29,348 1935       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-11-02 15:49:29,361 1948       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.9
2023-11-02 15:49:29,361 1948       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.1
2023-11-02 15:49:29,362 1949       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.20
2023-11-02 15:49:29,362 1949       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-11-02 15:49:29,362 1949       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-11-02 15:49:29,362 1949       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.99.1.el7.x86_64
No imports due to errors!

Didn't have this problem at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issue-1890818645 - I wonder what the difference is???

[will-moore]

Fixing broken mkngff Filesets....

Ran mkngff sql on 3 newly generated EBI Plates... Using Fileset IDs 21112.sql 21118.sql 21160.sql. These are still valid on idr-testing, even though they have been swapped (not linked to any images now), since they still have the correct prefix for mkngff to work.

Generated sql... See https://github.com/IDR/idr-utils/pull/56/commits/08437f407f8330949afed47e16e7e22362cc19e6

21160.sql - The new (broken) mkngff Fileset for this data is Fileset:ID 6313409. Updated 21160.sql with this ID and ran...

12:30

$ psql -U omero -d idr -h $DBHOST -f 21160.sql
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        6314843
(1 row)

Same with 21112.sql -> Fileset ID: 6313428...

$ psql -U omero -d idr -h $DBHOST -f 21112.sql
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        6314844
(1 row)

COMMIT

For 21118.sql this didn't need any update (original Fileset ID still valid). 12:35

$ psql -U omero -d idr -h $DBHOST -f 21118.sql
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        6314845
(1 row)

COMMIT

[will-moore]

Forgot to run symlink creation for those 3 plates... Using idr0015.csv to manually run these... NB: these use the original Filesets (which haven't been deleted) to get the fileset prefix...

$ omero mkngff symlink /data/OMERO/ManagedRepository 21112 "/bia-integrator-data/S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr"

Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr -> /bia-integrator-data/S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr

$ omero mkngff symlink /data/OMERO/ManagedRepository 21118 "/bia-integrator-data/S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr"
...
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533_mkngff/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr -> /bia-integrator-data/S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr

$ omero mkngff symlink /data/OMERO/ManagedRepository 21160 "/bia-integrator-data/S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr"
...
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953_mkngff/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr -> /bia-integrator-data/S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr

Checked all three links with e.g.

$ ls -alh /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953_mkngff/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr/
total 133K
drwxrwxrwx. 2 root root 4.0K Nov  6 19:52 .
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 ..
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 A
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 B
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 C
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 D
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 E
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 F
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 G
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 H
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 I
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 J
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 K
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 L
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 M
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 N
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 O
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 OME
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 P
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 Q
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 R
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 S
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 T
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 U
drwxrwxrwx. 2 root root 4.0K Oct 10 10:27 V
-rw-rw-rw-. 1 root root  32K Nov  6 19:52 .zattrs
-rw-rw-rw-. 1 root root   23 Nov  6 19:52 .zgroup

[will-moore]

Since we have a bunch of Plates on idr-testing that have failed memo file generation (via Seb's parallel script and via viewing in web), lets try on clean idr0125-pilot...

$ cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets/idr0015

(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
>   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        5288671
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/09/07-02-56.345
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/09/07-02-56.345_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/09/07-02-56.345_mkngff/0063750d-cd10-4759-8eca-c706a07b6693.zarr -> /bia-integrator-data/S-BIAD861/0063750d-cd10-4759-8eca-c706a07b6693/0063750d-cd10-4759-8eca-c706a07b6693.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/09/07-02-56.345
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/09/07-02-56.345_mkngff/0063750d-cd10-4759-8eca-c706a07b6693.zarr.bfoptions
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        5288672
(1 row)

Try viewing image http://localhost:1040/webclient/userdata/?show=image-1962455 (12:25)...

Image is viewable: Memo file took 69 mins (4178531 ms)

(base) bash-4.2$ grep 54c6252bfd9c -A 2 -B 2 /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-12-01 13:35:04,681 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr/OME/.METADATA.ome.xml.bfmemo (735340 bytes)
2023-12-01 13:35:04,681 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1701433526150] time[4178531] tag[loci.formats.Memoizer.setId]
2023-12-01 13:35:04,682 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr/OME/METADATA.ome.xml Series: 0

[will-moore]

Plates where memo file generation failed...

Image:1962455 Fileset:21112 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01
Image:1964831 Fileset:21118 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01
Image:1972041 Fileset:211160 TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01

When regenerating these, I didn't include the clientpath option...

(base) bash-4.2$ grep -c https *.sql
19237.sql:3195
19238.sql:3195
19269.sql:3195
19270.sql:3195
19301.sql:3195
19302.sql:3195
19308.sql:3195
19309.sql:3195
19471.sql:3195
20252.sql:3195
20302.sql:3195
20303.sql:3195
20305.sql:3195
20306.sql:3195
20901.sql:3195
20902.sql:3195
20903.sql:1890
21002.sql:3195
21051.sql:3195
21102.sql:3195
21103.sql:3195
21104.sql:3195
21105.sql:3195
21106.sql:3195
21107.sql:3195
21108.sql:3187
21109.sql:3187
21110.sql:3195
21111.sql:3195
21112.sql:0
21113.sql:3195
21114.sql:3195
21115.sql:3195
21116.sql:3195
21117.sql:3195
21118.sql:0
21119.sql:3195
21120.sql:3195
21121.sql:3195
21122.sql:3195
21123.sql:3195
21124.sql:3195
21125.sql:3195
21126.sql:3195
21151.sql:3195
21152.sql:3195
21153.sql:3195
21154.sql:3195
21155.sql:3195
21156.sql:3195
21157.sql:585
21158.sql:3195
21159.sql:3195
21160.sql:0
21161.sql:3195
21162.sql:3195
21163.sql:3195
21164.sql:3195
21165.sql:3195
21166.sql:3195
21167.sql:3195
21168.sql:3195
21169.sql:3195
21170.sql:3195
21171.sql:3195
21172.sql:3195
21173.sql:3195
21174.sql:3195
21175.sql:3195
21176.sql:3195
21177.sql:3195
21178.sql:3195
21179.sql:3195
21180.sql:3195
21182.sql:3195
21185.sql:3195
21188.sql:3195
21191.sql:3195
21194.sql:3195
21197.sql:3195
21200.sql:3195
21203.sql:3195
21218.sql:3195

Fixed in https://github.com/IDR/idr-utils/pull/56/commits/7bb3acf590ffbb7a00e09972f1c98c1f5b9bd81d using add_clientpath.py script in https://github.com/IDR/idr-utils/pull/56/commits/87e17ab8eddc379cda191ed84b29b1aec15fe5a8

[will-moore]

Tried viewing the other 2 Plates too: http://localhost:1040/webclient/userdata/?show=image-1964831 http://localhost:1040/webclient/userdata/?show=image-1972041

These are viewable on idr0125-pilot. So it looks like all these 3 Filesets that aren't viewable on idr-testing are OK, but something has prevented them being viewable there, possibly a failed attempt to create memo file before the symlink

[will-moore]

On idr-next, I omitted running the 3 sql scripts for regenerated Filesets above: https://github.com/IDR/idr-utils/pull/56/commits/125c4e559d5020dacbf359bc2bf7694dd86fb708

However, these are actually ready to go, since they are working on idr0125-pilot above (and now have clientpath links too).

Run on idr-next...

create idr0015a.csv:

idr0015/TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr,S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f,21118
idr0015/TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr,S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53,21160
idr0015/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr,S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c,21112

As wmoore, updated SECRET

for i in $(ls); do sudo sed -i 's/oldsecret/x1234xxxx-7b69-473b-98b2-xxxxxxxxx/g' $i; done

as omero-server

omero login
cd ngff_filesets
export IDRID=idr0015
for r in $(cat idr0015a.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
done

UPDATE 396
BEGIN
 mkngff_fileset
----------------
        6314434
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533_mkngff/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr -> /bia-integrator-data/S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/06/21-26-25.533_mkngff/d69df538-4684-4b32-8ded-d2f2af43af9f.zarr.bfoptions
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        6314435
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953_mkngff/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr -> /bia-integrator-data/S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/10/13-37-45.953_mkngff/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53.zarr.bfoptions
UPDATE 396
BEGIN
 mkngff_fileset 
----------------
        6314436
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr -> /bia-integrator-data/S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/06/00-58-20.828_mkngff/1a29207c-d50b-48b7-a7c0-54c6252bfd9c.zarr.bfoptions

[will-moore]

Viewing first image to trigger memo file... http://localhost:1080/webclient/?show=image-1962455

[dominikl]

Size of the dataset on S3:

Total Objects: 23498354
Total Size: 6.0 TiB

(aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 ls --summarize --human-readable --recursive --no-sign-request bia-integrator-data/S-BIAD861)

[sbesson]

Interestingly that's a increase by more than a factor 2 compared to the original format (2.5TB in studies.tsv).