Upload to BioStudies - Githubissues

Notes for upload to BioStudies workflow...

Create a zip per Fileset
Upload to BioStudies via FTP
Group Files into a Submission by listing in a csv File, and submit that
Zip contents will be uploaded to s3 and archived to Fire

See https://www.ebi.ac.uk/biostudies/submissions/help#help-files and https://www.ebi.ac.uk/bioimage-archive/submit/

Wanted to test with idr0054 data on pilot-zarr2-dev... Having logged-in and got FTP details from https://www.ebi.ac.uk/biostudies/submissions/files...

$ ssh pilot-zarr2-dev

$ sftp bsftp@ftp-private.ebi.ac.uk
write: Connection reset by peer
Couldn't read packet: Connection reset by peer

Trying from local machine:

$ sftp bsftp@ftp-private.ebi.ac.uk
kex_exchange_identification: read: Connection reset by peer
Connection reset by 193.62.193.142 port 22
Connection closed

Will try a different client...

Let's just try with Cyberduck and FileZilla from local machine... Grab the data...

$ rsync -rvP --progress pilot-zarr2-dev:/data/ngff/idr0054/ .
$ zip -r "Tonsil 1.ome.zarr.zip" "Tonsil 1.ome.zarr"
etc.

I can login with Cyberduck but can't work out how to use Cyberduck to upload to my "secret directory" on the server.

Manually uploaded zips via the web UI

https://www.ebi.ac.uk/biostudies/submissions/

Submitted -> https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD704

EDIT: Tried FileZilla and I can see how to upload files to the secret directory. I see the files there that I uploaded via the web UI.

From Matthew:

"Corresponding S3 URIs in our bucket for each NGFF fileset:"

Tonsil 2.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0
Tonsil 1.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr/0
Tonsil 3.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr/0

As discussed in IDR meeting today, I'll try to create a single goofys mount and look at how to use those mappings to create correct symlinks to the data on idr0125-pilot.

On idr0125-pilot:

$ sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

$ ls /bia-integrator-data/S-BIAD704
36cb5355-5134-4bdc-bde6-4e693055a8f9  3b4a8721-1a28-4bc4-8443-9b6e145efbe9  5583fe0a-bbe6-4408-ab96-756e8e96af55

Created: With https://github.com/IDR/idr-utils/pull/54/commits/2ce54a1eeeba661a22489c38d20b435c7d6807b9

# /tmp/idr0054_fileset_mappings.csv

Tonsil 2.ome.zarr,S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr
Tonsil 1.ome.zarr,S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr
Tonsil 3.ome.zarr,S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /bia-integrator-data --fileset-mappings /tmp/idr0054_fileset_mappings.csv --report

fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'}

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
Render Image 5025551
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
Render Image 5025552
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
Render Image 5025553
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr

Viewing Image after symlink update above fails,

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml

full stack trace

``` Traceback (most recent call last): File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner response = get_response(request) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 538, in wrapped retval = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 597, in wrapper context = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/views.py", line 1777, in load_metadata_preview rdefId = manager.image.getRenderingDefId() File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7944, in wrapped if not self._prepareRenderingEngine() \ File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8220, in _prepareRenderingEngine self._re = self._prepareRE(rdid=rdid) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8199, in _prepareRE re.load(ctx) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4859, in __call__ return self.handle_exception(e, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/webclient_gateway.py", line 2109, in handle_exception super(OmeroWebSafeCallWrapper, self).handle_exception(e, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4856, in __call__ return self.f(*args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx)) omero.ResourceError: exception ::omero::ResourceError { serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205) at jdk.internal.reflect.GeneratedMethodAccessor345.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy92.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202) at ome.services.RenderingBean.load(RenderingBean.java:417) at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy136.load(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy136.load(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316) at jdk.internal.reflect.GeneratedMethodAccessor1710.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy138.load_async(Unknown Source) at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248) at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223) at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) serverExceptionClass = ome.conditions.ResourceError message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml } ```

but the symlinks look OK:

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
total 0
drwxrwsr-x. 2 omero-server omero-server  31 May 29 13:00 .
drwxrwsr-x. 3 omero-server omero-server  50 May 18 09:50 ..
lrwxrwxrwx. 1 omero-server omero-server 109 May 29 13:00 Tonsil 1.ome.zarr -> /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr

$ cat /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil\ 1.ome.zarr/OME/METADATA.ome.xml 

<?xml version="1.0" encoding="UTF-8"?><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.o....

Reverted the symlinks to previous values and that fixed the rendering in webclient: NB: had to comment out # preview_image(conn, fileset_id, args) in the script since that crashed with the ResourceError above.

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /idr0054/zarr/ --report

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
fileset_dirs {}
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /idr0054/zarr/Tonsil 1.ome.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
fileset_dirs {}
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /idr0054/zarr/Tonsil 2.ome.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
fileset_dirs {}
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /idr0054/zarr/Tonsil 3.ome.zarr

I don't see any differences between what's in those locations:

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil\ 1.ome.zarr/
total 17K
drwxr-xr-x. 2 root root 4.0K May 18 09:06 .
drwxr-xr-x. 2 root root 4.0K May 18 09:18 ..
drwxr-xr-x. 2 root root 4.0K May 18 09:18 0
drwxr-xr-x. 2 root root 4.0K May 18 09:18 OME
-rw-r--r--. 1 root root   33 May 18 09:06 .zattrs
-rw-r--r--. 1 root root   23 May 18 09:06 .zgroup

$ ls -alh /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr
total 17K
drwxr-xr-x. 2 root root 4.0K May 19 16:50 .
drwxr-xr-x. 2 root root 4.0K May 29 11:11 ..
drwxr-xr-x. 2 root root 4.0K May 29 11:11 0
drwxr-xr-x. 2 root root 4.0K May 19 16:50 OME
-rw-r--r--. 1 root root   33 May 19 16:49 .zattrs
-rw-r--r--. 1 root root   23 May 19 16:49 .zgroup

Checked a bunch of other dirs too - couldn't see any differences.

The Aspera instructions on BioStudies:

The Aspera ascp command line client is distributed as part of the Aspera CLI high-performance transfer browser plug-in which can be downloaded from here You will need to execute the downloaded file and switch to the CLI folder, e.g.,

$ sh ~/Documents/ibm-aspera-cli-3.9.6.1467.159c5b1-mac-10.11-64-release.sh
$ cd ~/Applications/Aspera\ CLI/bin
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d <directory to upload> bsaspera_w@hx-fasp-1.ebi.ac.uk:<secret directory>

where - -P33001 and bsaspera_w@hx-fasp-1.ebi.ac.uk defines port, user and server for Aspera connection. - is the path of the directory to be uploaded. - looks like xx/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-xxxx and is shown above.

The download link there contains OSX but even if I remove that I only see OSX options to download. This will make it hard to install on the CentOS7 machines where we want to do the conversions etc.

@jburel pointed me at https://github.com/Euro-BioImaging/BatchConvert/ They upload to BIA via Aspera. The ascp cli tool is installed via conda:

https://github.com/Euro-BioImaging/BatchConvert/blob/c3a89791416babd2681b718ba0f6a043f9249d7a/dependencies/Dockerfile#L26

I followed that manually...

conda create --yes --name ascp2bia python=3.9
conda install -n ascp2bia -c hcc aspera-cli

  aspera-cli         hcc/osx-64::aspera-cli-3.9.6-hbec0bfc_0 None
You must agree to the license before using aspera-cli, http://www-03.ibm.com/software/sla/sladb.nsf/displaylis/2ABD3328689EBBFE8525830C007A5F38.

I copied the file at https://github.com/Euro-BioImaging/BatchConvert/blob/c3a89791416babd2681b718ba0f6a043f9249d7a/configs/asperaweb_id_dsa.openssh and saved it locally. This is normally included in the download of Aspera above.

$ conda activate ascp2bia
$ which ascp
/Users/wmoore/opt/anaconda3/envs/ascp2bia/bin/ascp

$ ascp -P33001 -i ~/.ssh/asperaweb_id_dsa.openssh -d Desktop/images/lif bsaspera_w@hx-fasp-1.ebi.ac.uk:**/*******************
Killed: 9

This might be failing because I haven't agreed to the license. Tried following http://www-03.ibm.com/software/sla/sladb.nsf/displaylis/2ABD3328689EBBFE8525830C007A5F38. but that redirects to License Information documents https://www.ibm.com/support/customer/csol/terms/#license-search with no clear info on what I need to do. Just looks like documents for reference, not any license agreement form?

Signed up for IBM account and logged-in, but that didn't change what I saw at links above. Downloaded according to instructions from BioStudies...

$ cd Documents/Aspera/
$ sh ibm-aspera-cli-3.9.6.1467.159c5b1-mac-10.11-64-release.sh

Installing IBM Aspera CLI

Installation into /Users/wmoore/Applications/Aspera CLI successful

Optional installation steps:

  To include aspera in your PATH, run this command (or add it to .bash_profile):
    export PATH=/Users/wmoore/Applications/Aspera\ CLI/bin:$PATH

  To install the man page, run the following command:
    export MANPATH=/Users/wmoore/Applications/Aspera\ CLI/share/man:$MANPATH

$ which ascp
/Users/wmoore/Applications/Aspera CLI/bin/ascp

$ cd ~/Applications/Aspera\ CLI/bin
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d ~/Desktop/images/lif/ bsaspera_w@hx-fasp-1.ebi.ac.uk:xx/xxxxxxxxxxxxxxxxxxxxxxx
20191025 Test FRET 585. 423, 426.lif                                                                                                                         100%   47MB  175Mb/s    00:02    
Completed: 48157K bytes transferred in 3 seconds
 (130222K bits/sec), in 1 file, 1 directory.

Success!

I see that you can choose the OS in the BioStudies page to get the correct link for e.g. Linux:

On idr0125-pilot:

$ wget https://ak-delivery04-mul.dhe.ibm.com/sar/CMA/OSA/08q6g/0/ibm-aspera-cli-3.9.6.1467.159c5b1-linux-64-release.sh

Installing IBM Aspera CLI

Installation into /home/wmoore/.aspera/cli successful

Optional installation steps:

  To include aspera in your PATH, run this command (or add it to .bash_profile):
    export PATH=/home/wmoore/.aspera/cli/bin:$PATH

  To install the man page, run the following command:
    export MANPATH=/home/wmoore/.aspera/cli/share/man:$MANPATH

$ cd /.aspera/cli/bin
$ ./ascp -P3..... # worked as above!

For idr0054 on pilot-idr0125, to debug symlinking to BioStudies s3, I updated the symlinks as above:

managed_repo_symlinks.py

``` $ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /bia-integrator-data --fileset-mappings /tmp/idr0054_fileset_mappings.csv --report Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/ Render Image 5025551 fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'} fs_contents ['Tonsil 1.ome.zarr'] Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/ Render Image 5025552 fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'} fs_contents ['Tonsil 2.ome.zarr'] Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/ Render Image 5025553 fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'} fs_contents ['Tonsil 3.ome.zarr'] Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr ```

$ tail -f /opt/omero/server/OMERO.server/var/log/Blitz-0.log

Resource Error Blitz.log

``` 2023-06-05 13:08:35,767 INFO [ org.perf4j.TimingLogger] (l.Server-0) start[1685970515763] time[3] tag[omero.call.success.ome.services.RenderingBean$7.doWork] 2023-06-05 13:08:35,767 INFO [ ome.services.util.ServiceHandler] (l.Server-0) Rslt: (ome.model.enums.RenderingModel:rgb:Id_1, ome.model.enums.RenderingModel:greyscale:Id_2) 2023-06-05 13:08:35,767 INFO [ ome.services.util.ServiceHandler] (l.Server-0) Executor.doWork -- ome.services.RenderingBean.getPixelBuffer[] 2023-06-05 13:08:35,767 INFO [ ome.services.util.ServiceHandler] (l.Server-0) Args: [null, InternalSF@441482912] 2023-06-05 13:08:35,768 INFO [ ome.security.basic.BasicEventContext] (l.Server-0) cctx: group=3 2023-06-05 13:08:35,769 INFO [ ome.security.basic.EventHandler] (l.Server-0) Auth: user=52,group=3,event=null(User),sess=a24936e0-1b10-40c9-b1a9-a95101791d8e 2023-06-05 13:08:35,770 INFO [ ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml 2023-06-05 13:08:35,984 DEBUG [ loci.formats.Memoizer] (l.Server-0) memo(lastModified=1684404160866) older than real(lastModified=1684515006000) 2023-06-05 13:08:36,038 INFO [ loci.formats.ImageReader] (l.Server-0) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml 2023-06-05 13:08:36,267 DEBUG [ loci.formats.Memoizer] (l.Server-0) start[1685970515772] time[494] tag[loci.formats.Memoizer.setId] 2023-06-05 13:08:36,268 ERROR [ ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml 2023-06-05 13:08:36,269 ERROR [ ome.io.nio.PixelsService] (l.Server-0) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1 at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79) at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124) at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898) at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571) at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205) at jdk.internal.reflect.GeneratedMethodAccessor345.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy92.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202) at ome.services.RenderingBean.load(RenderingBean.java:417) at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy136.load(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy136.load(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140) at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316) at jdk.internal.reflect.GeneratedMethodAccessor1710.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at omero.cmd.CallContext.invoke(CallContext.java:85) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy138.load_async(Unknown Source) at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248) at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223) at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalArgumentException: Invalid index: -1 at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328) at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137) at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116) at loci.formats.in.ZarrReader.initFile(ZarrReader.java:289) at loci.formats.FormatReader.setId(FormatReader.java:1443) at loci.formats.ImageReader.setId(ImageReader.java:849) at ome.io.nio.PixelsService$3.setId(PixelsService.java:869) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelFiller.setId(ChannelFiller.java:234) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293) at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) at loci.formats.Memoizer.setId(Memoizer.java:690) at ome.io.bioformats.BfPixelsWrapper.(BfPixelsWrapper.java:52) at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73) ... 82 common frames omitted 2023-06-05 13:08:36,271 INFO [ org.perf4j.TimingLogger] (l.Server-0) start[1685970515767] time[503] tag[omero.call.exception] 2023-06-05 13:08:36,271 INFO [ ome.services.util.ServiceHandler] (l.Server-0) Excp: ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml 2023-06-05 13:08:36,272 INFO [ org.perf4j.TimingLogger] (l.Server-0) start[1685970515759] time[512] tag[omero.call.exception] 2023-06-05 13:08:36,272 INFO [ ome.services.util.ServiceHandler] (l.Server-0) Excp: ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml 2023-06-05 13:08:36,279 INFO [ ome.services.util.ServiceHandler] (l.Server-9) Meth: interface ome.api.StatefulServiceInterface.close 2023-06-05 13:08:36,279 INFO [ ome.services.util.ServiceHandler] (l.Server-9) Args: () 2023-06-05 13:08:36,279 INFO [ org.perf4j.TimingLogger] (l.Server-9) start[1685970516279] time[0] tag[omero.call.success.ome.services.RenderingBean.close] 2023-06-05 13:08:36,279 INFO [ ome.services.util.ServiceHandler] (l.Server-9) Rslt: null 2023-06-05 13:08:36,279 INFO [ omero.cmd.SessionI] (l.Server-9) Unregistered servant:a24936e0-1b10-40c9-b1a9-a95101791d8e/5f20a8a7-a2ec-4982-82a3-f720bcc27a3comero.api.RenderingEngine(omero.api._RenderingEngineTie@88be2b14) 2023-06-05 13:08:36,281 INFO [o.services.sessions.SessionContext$Count] (l.Server-6) -Reference count: a24936e0-1b10-40c9-b1a9-a95101791d8e=0 2023-06-05 13:08:36,281 INFO [ omero.cmd.SessionI] (l.Server-6) cleanupSelf(ServiceFactoryI(session-33debeb7-69e1-44ef-95c6-fde7f78d243e/a24936e0-1b10-40c9-b1a9-a95101791d8e)). 2023-06-05 13:08:36,281 INFO [ omero.cmd.SessionI] (l.Server-6) Unregistered servant:a24936e0-1b10-40c9-b1a9-a95101791d8e/33debeb7-69e1-44ef-95c6-fde7f78d243eomero.api.IContainer(omero.api._IContainerTie@8a31d1da) ```

cc @dgault @sbesson Any ideas what might be causing this error? java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1 at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

Meeting with Matthew today:

[x] Need to create an IDR account for us to upload to BioStudies and submit. Use IDR e-mail account, which one?
[x] Upload zips to a directory within the "secret directory" on BioStudies. E.g. /idr0010/ - Can continue to add to this dir until submission is complete (doesn't need to all go in one FTP command).
[x] Need to list each .zarr.zip in the studies tsv for submission.
[x] We'll get a mapping from each image.zarr to uuid.zarr on s3, which we can use to generate correct symlinks.
[ ] Need to decide on consistent naming strategy for Submissions. E.g. idr0054 NGFF ....
[x] Create links of type: "Image data resource" in links section of Submission to Study in IDR, e.g. idr.openmicroscopy.org/webclient?show=project-1 so these can be searched for
[x] Our EBI s3 buckets should be deleted once not needed.
[ ] bioformats2raw images in idr0054 are 'greyscale' in omero settings?
[x] Need to list/check which studies have existing submissions in BioStudies - link to them from IDR submission
[ ] EBI will be able to delete my idr0054 submission - replace with new one from IDR account

Discussion with Josh and Matthew on slack about "zip zarrs": https://openmicroscopy.slack.com/archives/C01ABANKH8V/p1687170670769479 Safer to stick with zipping the image.zarr directory itself rather than trying to create a zip zarr by zipping the contents (less well tested at EBI).

IDR / idr-metadata

Upload to BioStudies #659