search

IDR / idr-metadata

Curated metadata for all studies published in the Image Data Resource
https://idr.openmicroscopy.org
14 stars 24 forks source link

idr0054-segura-tonsilhyperion S-BIAD800 #654

Open dominikl opened 1 year ago

dominikl commented 1 year ago

Export time: < 1 min / image Reimport time: a few min.

dominikl commented 1 year ago

Converted all three images on pilot-zarr2-dev in /data/ngff/idr0054

will-moore commented 1 year ago

Created bucket as at https://github.com/IDR/idr-metadata/issues/656

$ /home/wmoore/mc cp -r idr0054/ uk1s3/idr0054/zarr

# realised later this included idr0054/idr0054-segura-tonsilhyperion directory which we didn't want - removed:
./mc rm --recursive uk1s3/idr0054/zarr/idr0054-segura-tonsilhyperion --force

Looks good: https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0054/zarr/Tonsil+1.ome.zarr/0

Screenshot 2023-05-18 at 10 16 46

Mount on idr0125-pilot:

sudo mkdir /idr0054 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0054 /idr0054
$ ls /idr0054/zarr/
Tonsil 1.ome.zarr  Tonsil 2.ome.zarr  Tonsil 3.ome.zarr
will-moore commented 1 year ago

Downloaded metadata-only images as described.

Import command needed quotes since dir names contain spaces:

for dir in *; do   omero import --transfer=ln_s --depth=100 --name="${dir/.ome.zarr/}" --skip=all "$dir" --file "/tmp/$dir.log"  --errs "/tmp/$dir.err"; done

Created Filesets: 5287119, 5287120, 5287121 - Images: 14834093, 14834094, 14834095. e.g.

12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:06.171

Forgot to import into Dataset... Not displayed in webclient!

$ dataset=$(omero obj new Dataset name='idr0054_NGFF')
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834093
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834094
$ omero obj new DatasetImageLink parent=$dataset child=Image:14834095

Symlinks:

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:15351 /idr0054/zarr/ --report

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
Render Image 14834093
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /idr0054/zarr/Tonsil 1.ome.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
Render Image 14834094
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /idr0054/zarr/Tonsil 2.ome.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
Render Image 14834095
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /idr0054/zarr/Tonsil 3.ome.zarr

Images are viewable in webclient. 👍

Fileset swap...

$ python idr-utils/scripts/swap_filesets.py Dataset:5601 Dataset:15351 /tmp/idr0054_filesetswap.sql --report
SQL writing to /tmp/idr0054_filesetswap.sql
Swapping images between Fileset 1591301 and Fileset 5287119
Moving Image: 5025551 to Fileset: 5287119
Unlinking Image: 14834093
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME' where image in (select id from Image where fileset = 5287119);
Swapping images between Fileset 5286813 and Fileset 5287120
Moving Image: 5025552 to Fileset: 5287120
Unlinking Image: 14834094
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr/OME' where image in (select id from Image where fileset = 5287120);
Swapping images between Fileset 1591303 and Fileset 5287121
Moving Image: 5025553 to Fileset: 5287121
Unlinking Image: 14834095
UPDATE pixels SET name = 'METADATA.ome.xml', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr/OME' where image in (select id from Image where fileset = 5287121);
SQL output added to /tmp/idr0054_filesetswap.sql
$ PGPASSWORD=**** psql -U omero -d idr -h 192.168.10.102 -f /tmp/idr0054_filesetswap.sql 
UPDATE 1
UPDATE 1
UPDATE 1

Validate pixels - 

$ python idr-utils/scripts/check_pixels.py Dataset:5601 /tmp/check_pixels_idr0054.log 
$ tail -f  /tmp/check_pixels_idr0054.log 
Start: 2023-05-18 10:35:55.992703
Checking Dataset:5601
max_planes: 0
0/3 Check Image:5025551 Tonsil 1
1/3 Check Image:5025552 Tonsil 2
2/3 Check Image:5025553 Tonsil 3
End: 2023-05-18 10:38:32.605204

Deleted Dataset in webclient

will-moore commented 1 year ago

Uploaded .zip files, one for each Fileset to https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD704 in order for the unzipped zarrs to be made available on s3.

cc @mrmh2 @joshmoore @sbesson @dominikl @francesw

For some fields required for the submission I was adding info from the publication - not sure I got it all perfect, but hopefully nothing incorrect.

will-moore commented 1 year ago

Testing mkngff on idr0125-pilot

$ sudo -u root -s
$ source /opt/omero/server/venv3/bin/activate
$ pip install git+https://github.com/joshmoore/omero-mkngff.git
...
Resolved https://github.com/joshmoore/omero-mkngff.git to commit 68436532a8d2aa3ecb07164e08376a897389afa6
...
Successfully installed omero-mkngff-0.1.0.dev0
ls /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/
0  OME
$ sudo -u omero-server -s
$ source /opt/omero/server/venv3/bin/activate

$ omero mkngff setup

CREATE OR REPLACE FUNCTION mkngff_fileset(old_fileset bigint, uuid character varying, repo character varying, prefix character varying, info text[][])
  RETURNS integer AS
$BODY$
DECLARE
   fs_id integer;
   file_id integer;
BEGIN

    insert into fileset (id, permissions, templateprefix, creation_id, group_id, owner_id, update_id)
                 values (nextval('seq_fileset'), -120, prefix, 497, 0, 0, 497)
                 returning id into fs_id;

    for i in 1 .. array_upper(info, 1)
    loop

      insert into originalfile (id, permissions, mimetype, repo, path, name, creation_id, owner_id, group_id, update_id)
                    values (nextval('seq_originalfile'), -120, info[i][3], repo, info[i][1], uuid || info[i][2], 497, 0, 0, 500)
                    returning id into file_id;

      insert into filesetentry (id, permissions, clientpath, creation_id, update_id, owner_id, group_id, fileset, originalfile, fileset_index)
                    values (nextval('seq_filesetentry'), -120, 'unknown', 497, 497, 0, 0, fs_id, file_id, i-1);

    end loop;

    update image set fileset = fs_id where fileset = old_fileset;

    RETURN fs_id;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE;

$ omero mkngff sql 5287119 /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr --zarr_name "Tonsil 1.ome.zarr"

Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

begin;
    select new_fileset(
      5287119,
      'TBD',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr/',
      array[
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/', '09-50-49.857_converted', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/', 'Tonsil 1.ome.zarr', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '0', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '0', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/0/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '1', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/1/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '2', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/2/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '3', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/3/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '4', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/4/', '.zarray', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', 'OME', 'Directory'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zattrs', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zgroup', 'application/octet-stream'],
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', 'METADATA.ome.xml', 'application/octet-stream']
      ]::text[][]
    )
);
commit;

Tried pasting those into a psql terminal... Got errors due to an extra closing ) and had to replace new_fileset( with mkngff_fileset(.

That got me to....

idr=> begin;
BEGIN
idr=>     select mkngff_fileset(
idr(>       5287119,
idr(>       'TBD',
idr(>       'cdf35825-def1-4580-8d0b-9c349b8f78d6',
idr(>       'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr/',
idr(>       array[
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/', '09-50-49.857_converted', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/', 'Tonsil 1.ome.zarr', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', '0', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '0', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/0/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '1', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/1/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '2', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/2/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '3', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/3/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/', '4', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0/4/', '.zarray', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/', 'OME', 'Directory'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zattrs', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', '.zgroup', 'application/octet-stream'],
idr(>           ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/OME/', 'METADATA.ome.xml', 'application/octet-stream']
idr(>       ]::text[][]
idr(>     );
ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile (id, permissions, mimetype, repo, path, name, creation_id, owner_id, group_id, update_id)
                    values (nextval('seq_originalfile'), -120, info[i][3], repo, info[i][1], uuid || info[i][2], 497, 0, 0, 500)
                    returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 14 at SQL statement
idr=>     commit;
ROLLBACK

cc @joshmoore

joshmoore commented 1 year ago

Sorry, @will-moore. I'm still pushing commits (e.g. https://github.com/joshmoore/omero-mkngff/commit/2175a18290ee82b3ecf9d709e7558409c5457b74). Hopefully I'll have a commit ready for you today.

joshmoore commented 1 year ago

@will-moore: https://github.com/joshmoore/omero-mkngff/commit/fe4e8ae4da73ba46f0a4d0e679135454c7f002e5 is up-to-date and works as expected up to rendering where I get the Invalid index: -1 error.

Finding the secret UUID is still a bit complicated, but:

select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 10;

is a current starting point. Once you have that, the steps are:

SECRET=3a4d3a3f-ac98-4b1a-84ab-5534e364f181
micromamba create -n omero -c conda-forge -c ome omero-py bioformats2raw
micromamba activate omero
pip install 'omero-mkngff @ git+https://github.com/joshmoore/omero-mkngff@main'
omero login demo@localhost
echo series=2 >> a.fake.ini
touch a.fake
omero import a.fake.ini
bioformats2raw -p a.fake.ini a.ome.zarr
omero mkngff setup > setup.sql
omero mkgnff sql --secret=$SECRET 5287125 a.ome.zarr/
sudo -u postgres psql idr < setup.sql
sudo -u postgres psql idr < my.sql
sudo -u omero-server mkdir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/22/12-46-39.975_converted/

mv a.ome.zarr /tmp
ln -s /tmp/a.ome.zarr /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/22/12-46-39.975_converted/a.ome.zarr
omero render test Image:14834721 # Failing here
will-moore commented 11 months ago

The data for my first BioStudies submission above is at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD704.html

You can get the mapping of zarr -> uuid by running this in the browser console:

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "td:nth-child(1)", $this).text();
    let zarrzip = $( "td:nth-child(3)", $this).text();
    csv += `${zarrzip},${uid}\n`
});
document.write(`<code><pre>${csv}</pre></code>`);

creates:

Tonsil 3.ome.zarr.zip,3b4a8721-1a28-4bc4-8443-9b6e145efbe9
Tonsil 2.ome.zarr.zip,36cb5355-5134-4bdc-bde6-4e693055a8f9
Tonsil 1.ome.zarr.zip,5583fe0a-bbe6-4408-ab96-756e8e96af55
will-moore commented 11 months ago

IDR submission is at https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD800

will-moore commented 11 months ago

Data at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD800.html

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrname = $( "td:nth-child(3)", $this).text().replace(".zip", "");
    csv += `${zarrname},${uid}\n`
});
console.log(csv);

Gives:

idr0054/Tonsil 1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39
idr0054/Tonsil 2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573
idr0054/Tonsil 3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3
will-moore commented 10 months ago

mkngff test... Removing white-space and manually adding Fileset IDs... idr0054.csv:

idr0054/Tonsil_1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39,5287366
idr0054/Tonsil_2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573,5287367
idr0054/Tonsil_3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3,5287121
for r in $(cat idr0054.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3)
  omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857_converted for fileset 5287366
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr -> /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18 // 09-51-02.628_converted for fileset 5287367
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr -> /bia-integrator-data/S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18 // 09-51-13.702 for fileset 5287121
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702_mkngff/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr -> /bia-integrator-data/S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr
for r in $(cat idr0054.csv); do
  fsid=$(echo $r | cut -d',' -f3)
  psql -U omero -d idr -h $DBHOST -f "$fsid.sql"
done

Screenshot 2023-08-29 at 16 40 06

will-moore commented 7 months ago 
(venv3) [wmoore@test120-omeroreadonly-1 scripts]$ python check_pixels.py Project:701 --max-planes=sizeC
Start: 2023-12-06 20:40:07.302346
Checking Project:701
max_planes: sizeC
max_images: 0
0/3 Check Image:5025551 Tonsil 1
1/3 Check Image:5025552 Tonsil 2
2/3 Check Image:5025553 Tonsil 3
End: 2023-12-06 20:44:17.278682