Document mkngff workflow

will-moore commented 1 year ago

NB This workflow has been moved to https://github.com/IDR/mkngff_upgrade_scripts (main README)

Once a submission has been processed by BioStudies, it will become available at a URL like: https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html

We need to harvest the uuids from the links in the "Viewable Images" table. We can do this with the following JavaScript code, pasted into the Console tab of the browser dev tools:

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrname = $( "td:nth-child(3)", $this).text().replace(".zip", "");
    csv += `${zarrname},${uid}\n`
});
console.log(csv);

Which will print something like this:

idr0051/180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr,S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr,S-BIAD815/b2633930-86b0-489e-a845-d2a7afe6ff15
idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr,S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120
idr0051/2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr,S-BIAD815/e12a8e2a-4fce-4579-a78b-b0c4597c3ada

That CSV is a table of filesetName.ome.zarr, UUID. We need to add the Fileset IDs from IDR to that table, using idr-util scripts from https://github.com/IDR/idr-utils/pull/56 That PR contains a file idr_filesets.csv which contains Fileset ID, filesetName.ome.zarr from IDR. It also contains a script to take the csv from above and add the appropriate Fileset IDs (from idr_filesets.csv).

Checkout the idr-utils branch of that PR. This can be done on a local machine. Copy the csv generated by JavaScript above and save it into a file like idr-utils/scripts/ngff_filesets/idr0051.csv. You will see some examples included in that PR. Then run the script, passing in the IDR ID...

$ cd idr-utils/scripts/ngff_filesets
$ python parse_bia_uuids.py idr0051

This will update the csv file you just created, adding in the Fileset IDs to a new 3rd column.

Now we want to use that data with omero-mkngff. We need to do everything as the omero-server user since we'll want to be able to create symlinks from the ManagedRep.

E.g. working on idr0138-pilot...

$ sudo -u omero-server -s

Created conda environment created as omero-server user, e.g. mkngff and installed omero-py and omero-mkngff

conda create -n mkngff -c conda-forge -c ome omero-py bioformats2raw
conda activate mkngff
pip install 'omero-mkngff @ git+https://github.com/IDR/omero-mkngff@main'

Get Database password (and host) needed for psql, and set these to env variables. Also set variable for $IDRID so you can copy and paste other commands from below...

export IDRID=idr0012
export OMERODIR=/opt/omero/server/OMERO.server
omero config get | grep omero.db.host
$ export DBHOST=192.168.10.231
omero config get --show-password | grep omero.db.pass
export PGPASSWORD=[********]

Use psql to get SECRET (last session ID). NB: for pilot servers we only have 1 process (as in this example). For other servers, update the 1 to 3 in this psql command:

psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"
                 uuid                 
--------------------------------------
 8add790d-7855-46f6-8239-c6a72937d572
(1 row)

export SECRET=8add790d-7855-46f6-8239-c6a72937d572

Copy the contents of idr0051.csv table from above (contains Fileset ID and UUID) and create a copy of the csv in the omero-server user's home dir...

$ cd
$ vi $IDRID.csv   # paste in the csv contents from above

Now we can read that csv and create an sql file for each Fileset (named FILESET_ID.sql). In the loop below, biapath is like S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97 and uuid is like 51afff7c-eed4-44b4-95c7-1437d8807b97.

The BIA s3 repository should be mounted under /bia-integrator-data:

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

Check that e.g. $ ls /bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr will give you 0 OME Theomero mkngff command below also creates the symlinks we need, from the ManagedRepository to the s3-mounted data (if they don't already exist).

# first output sql functions and login...
omero mkngff setup > setup.sql
omero login

$ mkdir -p $IDRID
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

# IF YOU WANT TO EXECUTE SQL IMMEDIATELY... include $SECRET and create symlinks...
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql" --bfoptions
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26 // 10-39-49.639 for fileset 604306
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639_converted/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639_converted/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr -> /bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr
BEGIN
 mkngff_fileset 
----------------
        5811532
(1 row)
COMMIT
...

Running sql on a different server (using saved sql)

Zip and copy sql to a different server. Unzip and update the SECRET in all sql files, getting current $SECRET as above The replace didn't work using $SECRET etc in the regex, so just use actual values... SECRETUUID is the default placeholder if you didn't use --secret option to create sql.

$ for i in $(ls); do sed -i 's/SECRETUUID/fc5d3566-eea0-412c-849e-daa6d3c6bfcc/g' $i; done

We want to execute all sql, using the csv, and also to use omero mkngff to do just the symlink creation...

$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr -> /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr

Now we can try viewing the images in webclient. NB: sometimes this can take a while for the memo file to be regenerated. To check on the timings you can use unique string from the fileset name

grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "46.368_mkngff"

2023-08-29 12:21:51,993 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-05/11/22-57-46.368_mkngff/HT20.ome.zarr/OME/.METADATA.ome.xml.bfmemo (3838714 bytes)
2023-08-29 12:21:51,993 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1693309192879] time[2519114] tag[loci.formats.Memoizer.setId]
2023-08-29 12:21:51,995 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-05/11/22-57-46.368_mkngff/HT20.ome.zarr/OME/METADATA.ome.xml Series: 0

E.g. 2519114 ms is 42 minutes.

joshmoore commented 1 year ago

Where do we get the paths for making the mkdir command?

The output sql shows the path.

I see that all the examples have _converted in them. Is this part of the API?

It's hard-coded in the script at the moment but otherwise pretty arbitrary.

The // in the output is also arbitrary but you are right we could print the target directory there. 👍🏼

will-moore commented 1 year ago

On idr0125-pilot...

Created conda environment mkngff and installed omero-py and omero-mkngff as above...

Get Database password...

export OMERODIR=/opt/omero/server/OMERO.server
omero config get --show-password

Use psql to get SECRET (last session ID):

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102
idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 688fcb80-91f1-46ec-92ff-0547a1f6553x
(1 row)

export SECRET=688fcb80-91f1-46ec-92ff-0547a1f6553x
omero mkngff setup > setup.sql
omero login
omero mkngff sql --secret=$SECRET 5287119 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

SQL has paths like demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//idr0054/zarr/Tonsil 1.ome.zarr/OME/

Create symlinks...

$ sudo -u omero-server mkdir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted
$ sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"

$ ls "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"
0  OME

will-moore commented 1 year ago

$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f setup.sql
CREATE FUNCTION

$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql 
BEGIN
psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK

I don't know if that is a permissions issue? I'm running the sql commands there a bit differently from instructions above.

Trying those instructions...

$ sudo -u postgres psql idr < setup.sql
could not change directory to "/home/wmoore": Permission denied
CREATE FUNCTION

Maybe that's also a permissions issue, but don't know what's needed there?

will-moore commented 1 year ago

Looking at the sql it has:

begin;
    select mkngff_fileset(
      5287119,
      '3a4d3a3f-ac98-4b1a-84ab-XXXXXXXXXXX',
      'cdf35825-def1-4580-8d0b-XXXXXXXXXXX',
      'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/',
      array[
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//idr0054/zarr/Tonsil 1.ome.zarr/', '.zattrs', 'application/octet-stream'],
...

the 3a4d3a3f... value is the $SECRET that still matches:

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;

but I don't know what the cdf35825-def1-4580-8d0b-XXXXXXXXXXX value is?

joshmoore commented 1 year ago

but I don't know what the cdf35825-def1-4580-8d0b-XXXXXXXXXXX value is?

That's the repositoryID.

joshmoore commented 1 year ago

$ sudo -u postgres psql idr < setup.sql
could not change directory to "/home/wmoore": Permission denied
CREATE FUNCTION

This is just because sudo is trying to change the directory and can't. If you have correct SQL, it wasn't fatal.

joshmoore commented 1 year ago

psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key

This will mean it's the wrong secret key.

will-moore commented 1 year ago

With the instructions you provided for getting the secret key (at the top) it prints lots of info - including 3 session keys. I have tried to create a less-ambiguous way of getting the secret key. Is this the correct way to get the secret key?

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;

joshmoore commented 1 year ago

Part of the problem is that it's different on the pilots than it is from main servers. (It's based on the number of java processes that are started!)

will-moore commented 1 year ago

OK, so it's different on pilots from other servers. So what should I do?

Trying this on idr0125-pilot...

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 3a4d3a3f-ac98-4b1a-84ab-5534e364f181
(1 row)

$ export SECRET=3a4d3a3f-ac98-4b1a-84ab-5534e364f181

Followed same steps as above and got the same error...

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql 
BEGIN
psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key

So this seems to be not the right way to get the secret on the pilots. How do I need to adapt this to work on the pilots?

joshmoore commented 1 year ago

Sorry. The "limit 3" in the above is for a full server (with Indexer and PixelData process). Changing that to limit 1 should do the trick:

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;

will-moore commented 1 year ago

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 8add790d-7855-46f6-8239-c6a72937d572
(1 row)

export SECRET=8add790d-7855-46f6-8239-c6a72937d572
omero login
omero mkngff sql --secret=$SECRET 5287119 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql
BEGIN
 mkngff_fileset 
----------------
        5287366
(1 row)

COMMIT

will-moore commented 1 year ago

Getting ResourceError when viewing the image. Need to update ZarrReader to latest merge build. Following steps at https://github.com/IDR/idr-metadata/issues/643#issuecomment-1543728125

Get link from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/499/default/artifact/bio-formats-build/ZarrReader/target/

sudo -u omero-server -s
ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1080:localhost:80

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/499/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

$ mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
$ rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar
$ cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
$ rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
$ cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/

# as regular user
$ sudo service omero-server restart

But I still see ResourceError when viewing the image...

ResourceError

``` Traceback (most recent call last): File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner response = get_response(request) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 538, in wrapped retval = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 597, in wrapper context = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/views.py", line 1777, in load_metadata_preview rdefId = manager.image.getRenderingDefId() File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7944, in wrapped if not self._prepareRenderingEngine() \ File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8220, in _prepareRenderingEngine self._re = self._prepareRE(rdid=rdid) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8199, in _prepareRE re.load(ctx) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4859, in __call__ return self.handle_exception(e, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/webclient_gateway.py", line 2109, in handle_exception super(OmeroWebSafeCallWrapper, self).handle_exception(e, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4856, in __call__ return self.f(*args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx)) omero.ResourceError: exception ::omero::ResourceError { serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205) at jdk.internal.reflect.GeneratedMethodAccessor350.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy92.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202) at ome.services.RenderingBean.load(RenderingBean.java:417) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy121.load(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy121.load(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy123.load_async(Unknown Source) at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248) at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223) at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) serverExceptionClass = ome.conditions.ResourceError message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml } ```

joshmoore commented 1 year ago

The contents of your _converted directory are:

[jamoore@pilot-idr0125-omeroreadwrite log]$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/
Tonsil 1.ome.zarr

but the error messages says:

2023-08-08 13:38:15,237 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Excp:    ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml

i.e. there are two extra directory paths: "idr0054/zarr/"

will-moore commented 1 year ago

# remove symlink and re-create...
rm "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"
sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/
sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr"

Looks better...

$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil\ 1.ome.zarr/
0  OME

But I still see:

serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
    at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907)
    at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)

joshmoore commented 1 year ago

Hmmm..... did you possibly run the script twice? In the server log I see:

Caused by: ome.conditions.SecurityViolation:
file demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/0/.zattrs
does not have a unique readable fileset

will-moore commented 1 year ago

Trying with a different Fileset on idr0125-pilot...

$ omero mkngff sql --secret=$SECRET 5287120 "/idr0054/zarr/Tonsil 2.ome.zarr/" > idr0054_2.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18 // 09-51-02.628 for fileset 5287120

Extrapolate the dirs we need for symlinks... demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/

sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/
sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 2.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr"

Run sql...

$ psql -U omero -d idr -h 192.168.10.102 -f setup.sql
CREATE FUNCTION
$ psql -U omero -d idr -h 192.168.10.102 -f idr0054_2.sql
BEGIN
 mkngff_fileset 
----------------
        5287367
(1 row)

COMMIT

Rendering image...

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/OME/METADATA.ome.xml

will-moore commented 1 year ago

Blitz log has same error as before:

2023-08-16 10:04:34,084 ERROR [                ome.io.nio.PixelsService] (l.Server-6) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/.zattrs does not have a unique readable fileset
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

will-moore commented 1 year ago

Trying different server: idr0138-pilot... install miniconda etc...

$ wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh
$ bash Miniconda3-py39_4.10.3-Linux-x86_64.sh
$ source /home/wmoore/.bashrc
$ conda create -n mkngff -c conda-forge -c ome omero-py bioformats2raw
$ conda activate mkngff
$ pip install 'omero-mkngff @ git+https://github.com/joshmoore/omero-mkngff@main'

$ psql -U omero -d idr -h 192.168.10.231
psql (11.16)
Type "help" for help.

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 e1571d5f-ff16-4e79-b0d3-0352d4c4df44
(1 row)

Need to mount BIA s3 on idr0138-pilot....

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

$ omero mkngff sql --secret=$SECRET 1591301 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15 // 15-27-40.023 for fileset 1591301

demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023

sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/
$ sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr"

$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil\ 1.ome.zarr/

$ psql -U omero -d idr -h 192.168.10.231 -f setup.sql
$ psql -U omero -d idr -h 192.168.10.231 -f idr0054.sql
BEGIN

 mkngff_fileset 
----------------
        5811532
(1 row)

COMMIT

Rendering fails - expected since we don't have latest ZarrReader...

2023-08-16 10:54:40,374 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1

Update ZarrReader on idr0138-pilot:

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/default/507/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar

$ rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar
rm: cannot remove ‘/opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar’: No such file or directory

Hmm - seems that idr0138 doesn't have ZarrReader - can we just add it?

sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
sudo service omero-server restart
Redirecting to /bin/systemctl restart omero-server.service

Try rendering again - now we get same error as on idr0125-pilot:

2023-08-16 11:00:20,702 ERROR [                ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/.zattrs does not have a unique readable fileset
    at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

will-moore commented 1 year ago

Testing execute sql from one server on another with idr0033. Sql generated on idr-testing, copied to idr0125-pilot, updated SECRET etc (also symlinks created).

$ psql -U omero -d idr -h $DBHOST -f 23119.sql 
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5287787
(1 row)

COMMIT

Checking image... http://localhost:1040/webclient/?show=image-3249534

(base) bash-4.2$ grep -A 2 "458_mkngff/9c1a5073" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 14:54:41,906 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/23-50-06.458_mkngff/9c1a5073-3ebd-4e04-894c-900022e554e1.zarr/OME/.METADATA.ome.xml.bfmemo (10558639 bytes)
2023-09-21 14:54:41,906 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1695303204087] time[4877818] tag[loci.formats.Memoizer.setId]
2023-09-21 14:54:41,906 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/23-50-06.458_mkngff/9c1a5073-3ebd-4e04-894c-900022e554e1.zarr/OME/METADATA.ome.xml Series: 9

4877818 ms is 81 minutes

will-moore commented 1 year ago

Test whether we need to add Directory rows to the NGFF Filesets.

On idr0125-pilot, using the sql from idr-testing, remove rows with Directory from sql...

$ sed -i '/Directory/d' 23120.sql

$ psql -U omero -d idr -h $DBHOST -f 23120.sql
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5287789
(1 row)
COMMIT

http://localhost:1040/webclient/?show=well-1316985 ...

This is viewable with

File count: 21525 compared with File count: 39206 for the Filesets above (containing Directories).

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "252_mkngff/7e7bc" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 17:40:22,272 DEBUG [                   loci.formats.Memoizer] (.Server-10) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/.METADATA.ome.xml.bfmemo (10558531 bytes)
2023-09-21 17:40:22,272 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1695312725659] time[5296612] tag[loci.formats.Memoizer.setId]
2023-09-21 17:40:22,273 INFO  [                ome.io.nio.PixelsService] (.Server-10) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/METADATA.ome.xml Series: 117
--
2023-09-21 18:57:56,345 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/.METADATA.ome.xml.bfmemo (10558487 bytes)
2023-09-21 18:57:56,345 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1695317413272] time[5263072] tag[loci.formats.Memoizer.setId]
2023-09-21 18:57:56,345 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/METADATA.ome.xml Series: 117

5296612 ms is 88 minutes.

will-moore commented 1 year ago

Testing https://github.com/yandex-cloud/geesefs as an alternative to goofys...

On idr0125-pilot...


$ wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64
$ chmod +x geesefs-linux-amd64
$ sudo mv geesefs-linux-amd64 /opt/geesefs

# similar command and options as goofys...

$ sudo mkdir /bia-integrator-geesefs
$ sudo /opt/geesefs --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-geesefs
s3.INFO anonymous bucket detected
main.INFO File system has been successfully mounted.

$ ls /bia-integrator-geesefs/S-BIAD861
0063750d-cd10-4759-8eca-c706a07b6693
...

will-moore commented 1 year ago

Installed geesefs on idr-testing:omeroreadwrite as above.

Try running mkngff there using geesefs compared with using goofys. As wmoore user...

screen -S mkngff
cd
mkdir idr0013_geese
wget https://raw.githubusercontent.com/IDR/idr-utils/cac35aa0d1731afb5db0ab6b60e10bdf03c591fd/scripts/ngff_filesets/idr0013.csv

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid "/bia-integrator-geesefs/$biapath/$uuid.zarr" > "idr0013_geese/$fsid.sql"
done

will-moore commented 12 months ago

mkngff is still running on idr0013.csv on idr-testing above... Hasn't failed after 337 sql generated...

[wmoore@test120-omeroreadwrite ~]$ ls idr0013_geese/ | wc
    337     337    3370

Since 1pm on 6th Oct approx 67 hours -> 12 minutes per sql on average, although the timings are quite variable... E.g. range of 27 minutes to 3 minutes in sample below...

[wmoore@test120-omeroreadwrite ~]$ for r in $(cat idr0013.csv); do   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]');   ls -alh idr0013_geese/$fsid.sql; done
...
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 03:37 idr0013_geese/18781.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 03:41 idr0013_geese/18572.sql
-rw-rw-r--. 1 wmoore wmoore 215K Oct  9 03:52 idr0013_geese/18942.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 04:31 idr0013_geese/18475.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 04:58 idr0013_geese/18509.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:07 idr0013_geese/18785.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:21 idr0013_geese/18583.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:44 idr0013_geese/18680.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 05:50 idr0013_geese/18378.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 06:25 idr0013_geese/18357.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 06:47 idr0013_geese/18536.sql
-rw-rw-r--. 1 wmoore wmoore 576K Oct  9 06:50 idr0013_geese/18832.sql
-rw-rw-r--. 1 wmoore wmoore 0 Oct  9 06:50 idr0013_geese/18653.sql

IDR / omero-mkngff

Document mkngff workflow #2