Open will-moore opened 11 months ago
Import done.
Need to update symlinks for 11151-11219.
Since these were imported with ZarrReader.jar
from performance PR, we have extra /data/ngff
in path, which needs correct commit of managed_repo_symlinks
PR:
$ sudo -Es git checkout 2888b21c2751421ecf24a
...
2888b21... TEMP add /ngff/idr0125 to templatePrefix for each fileset
As omero-server... 10:38...
for i in {11151..11219}; do
echo "$i"
python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done
EDIT: 11:00... leave running..
Fileset: 5288583 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-39/2023-11/08/14-55-46.675/ngff/idr0125
Render Image 16296891
fs_contents ['SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-39/2023-11/08/14-55-46.675/ngff/idr0125/SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr
11164
fileset_dirs {}
Fileset: 5288584 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-50/2023-11/08/17-29-10.806/ngff/idr0125
Render Image 16298674
Tried browsing webclient in preparation for thumbnailing... Got DatabaseBusyException
in webclient.
symlinking got to...
preview_image(conn, fileset_id, args)
File "idr-utils/scripts/managed_repo_symlinks.py", line 62, in preview_image
image.renderJpeg()
File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7949, in wrapped
except ctx.ignoreExceptions:
TypeError: catching classes that do not inherit from BaseException is not allowed
11177
fileset_dirs {}
Can't login - need $ sudo service omero-server restart
...
12:41...
for i in {11176..11219}; do echo "$i"; python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report; done
...
11219
fileset_dirs {}
Fileset: 5288639 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2023-11/14/10-23-13.402/ngff/idr0125
Render Image 16491466
fs_contents ['SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2023-11/14/10-23-13.402/ngff/idr0125/SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr
completed (last link created at) 13:53. 70 mins for 43 symlinks
Browsing webclient to check links before running render...
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328)
... 57 common frames omitted
2023-11-16 16:13:48,736 INFO [ org.perf4j.TimingLogger] (.Server-38) start[1700151134730] time[94006] tag[omero.call.exception]
2023-11-16 16:13:48,737 WARN [ ome.services.util.ServiceHandler] (.Server-38) CannotCreateTransactionException thrown.
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBus
yException: Cannot acquire connection
This is without running anything else just now...
Restart server again...
Browsing a different plate on webclient - again...
Ice.UnknownException: exception ::Ice::UnknownException
{
unknown = ome.conditions.DatabaseBusyException: cannot create transaction
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
After leaving the server for a few days...
for i in {11053..11058}; do echo "$i"; python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report; done
Tried again viewing an Image from the 3rd Plate: Plate: 11053, SQ00014814__2016-05-23T17_24_56-Measurement1
.
I don't have a record of creating symlinks for that Plate, but the symlink and chunks seem to be there:
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/
total 0
drwxrwsr-x. 2 omero-server omero-server 67 Oct 31 09:57 .
drwxrwsr-x. 4 omero-server omero-server 94 Oct 30 17:11 ..
lrwxrwxrwx. 1 omero-server omero-server 142 Oct 31 09:57 SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr
$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/1/0/0/0/0/0/0
total 4.3M
drwxr-xr-x. 2 root root 4.0K Dec 7 2022 .
drwxr-xr-x. 2 root root 4.0K Dec 7 2022 ..
-rw-r--r--. 1 root root 2.0M Dec 7 2022 0
-rw-r--r--. 1 root root 2.1M Dec 7 2022 1
-rw-r--r--. 1 root root 239K Dec 7 2022 2
However, images from the Plate are not viewable. Tried browsing the next 3 plates
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource dfce81f1-9a82-4a7c-b00e-4452ee77a8ba still empty after 30s wait time
at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:319)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328)
... 25 common frames omitted
2023-11-20 12:41:30,742 ERROR [ o.s.blitz.fire.PermissionsVerifierI] (.Server-34) Exception thrown while checking password for:public
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy85.doWork(Unknown Source)
at ome.services.util.Executor$Impl.executeSql(Executor.java:549)
at ome.services.blitz.fire.Ring.checkPassword(Ring.java:270)
at ome.services.blitz.fire.PermissionsVerifierI.checkPermissions(PermissionsVerifierI.java:97)
at Glacier2._PermissionsVerifierDisp.___checkPermissions(_PermissionsVerifierDisp.java:114)
at Glacier2._PermissionsVerifierDisp.__dispatch(_PermissionsVerifierDisp.java:149)
at IceInternal.Incoming.invoke(Incoming.java:221)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
at Ice.ConnectionI.message(ConnectionI.java:1056)
at IceInternal.ThreadPool.run(ThreadPool.java:395)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: ome.conditions.DatabaseBusyException: Cannot acquire connection
at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:113)
at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
... 20 common frames omitted
2023-11-20 12:41:31,872 INFO [ ome.services.blitz.fire.SessionManagerI] (.Server-34) Found session locally: 416569f8-07bc-44df-afdd-bea2d004cfa1
2023-11-20 12:41:31,873 INFO [ ome.services.blitz.fire.SessionManagerI] (.Server-34) Rejoining session ServiceFactoryI(session-ba637349-1805-4f29-a971-4e26340c25c1/416569f8-07bc-44df-afdd-bea2d004cfa1) (agent=OMERO.web)
2023-11-20 12:41:31,878 INFO [ omero.cmd.SessionI] (.Server-35) Added servant to adapter: 416569f8-07bc-44df-afdd-bea2d004cfa1/ba637349-1805-4f29-a971-4e26340c25c1omero.api.IConfig(omero.api._IConfigTie@a7b89013)
Also still seeing ZarrReader logs...
...
2023-11-20 12:54:19,871 ERROR [ loci.formats.FormatHandler] (l.Server-0) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
2023-11-20 12:54:19,871 ERROR [ loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
2023-11-20 12:54:19,871 ERROR [ loci.formats.FormatHandler] (l.Server-1) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
...
Returning to idr0125-pilot again (since updating the ZarrReader to test idr0015 NGFF data) it looks like all the idr0125 data imported above is not viewable, possibly due to memo file regeneration.
Since we now have ZarrReader-0.4.0 released now with the NGFF performance improvements, lets install that and start a fresh import of all data...
wget https://artifacts.openmicroscopy.org/artifactory/ome.releases/ome/OMEZarrReader/0.4.0/OMEZarrReader-0.4.0.jar
...update, retstart etc...
sudo -Es git checkout -- screenA/idr0125-screenA-plates.tsv
Renamed previous idr0125 Screen in webclient, so fresh one should get created... bulk.yml
still commented out the clientpath check
omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231214.log --errs /tmp/idr0125_20231214.err
Import failed...
2023-12-14 10:19:52,383 687 [ main] INFO ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-12-14 10:19:52,425 729 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 6.12.0 revision: 9d2f3cc080ff771569ca08fb7b3fe3558b8396f2 date: 14 February 2023
2023-12-14 10:19:52,511 815 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-12-14 10:19:52,516 820 [ main] INFO formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-12-14 10:19:52,516 820 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-12-14 10:19:52,516 820 [ main] INFO formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-12-14 10:19:52,516 820 [ main] INFO formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-12-14 10:19:52,516 820 [ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-12-14 10:19:52,517 821 [ main] INFO formats.importer.cli.CommandLineImporter - Setting output format: yaml
2023-12-14 10:19:52,517 821 [ main] INFO formats.importer.cli.CommandLineImporter - Setting parallel upload: 10
2023-12-14 10:19:52,517 821 [ main] INFO formats.importer.cli.CommandLineImporter - Setting parallel fileset: 10
2023-12-14 10:19:52,517 821 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-12-14 10:19:55,317 3621 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-12-14 10:23:27,435 215739 [ main] INFO ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 1 group(s) with 1 call(s) to setId in 190658ms. (212117ms total) [0 unknowns]
2023-12-14 10:23:29,035 217339 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-12-14 10:23:31,174 219478 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-12-14 10:23:31,991 220295 [ main] INFO ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-12-14 10:23:32,003 220307 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-12-14 10:23:32,004 220308 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-12-14 10:23:32,004 220308 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-12-14 10:23:32,004 220308 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-12-14 10:23:32,004 220308 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-12-14 10:23:32,004 220308 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-12-14 10:23:32,805 221109 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Screen:name:idr0125-way-cellpainting/screenA
2023-12-14 10:23:33,180 221484 [2-thread-1] INFO ome.formats.importer.ImportLibrary - Import target specifies container: Screen:3501
2023-12-14 10:23:33,186 221490 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-12-14 11:27:50,970 4079274 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-12-14 11:27:51,022 4079326 [3-thread-3] INFO s.importer.transfers.SymlinkFileTransfer - Transferring /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zattrs...
2023-12-14 11:27:51,022 4079326 [3-thread-2] INFO s.importer.transfers.SymlinkFileTransfer - Transferring /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/0/.zarray...
...
2023-12-14 11:36:56,558 4624862 [3-thread-9] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/8/.zattrs
2023-12-14 11:36:56,566 4624870 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zgroup
2023-12-14 11:36:56,927 4625231 [3-thread-4] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/.zgroup
2023-12-14 11:46:15,888 5184192 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-12-14 11:46:16,286 5184590 [2-thread-1] ERROR ome.formats.importer.ImportLibrary - Error on import
Ice.UnmarshalOutOfBoundsException: null
at IceInternal.BasicStream.readSize(BasicStream.java:632)
at IceInternal.BasicStream$EncapsDecoder11.readObject(BasicStream.java:3543)
at IceInternal.BasicStream.readObject(BasicStream.java:2262)
...
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-12-14 11:46:16,287 5184591 [2-thread-1] INFO ome.formats.importer.ImportLibrary - Continuing after error
==> Summary
11157 files uploaded, 0 filesets created, 0 images imported, 0 errors in 1:22:43.987
Looking at Blitz logs around then doesn't show much...
grep "2023-12-14 11:46:16" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.* | less
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,142 INFO [ org.perf4j.TimingLogger] (.Server-13) start[1702554376139] time[3] tag[omero.call.success.ome.services.blitz.impl.ServiceFactoryI$3.doWork]
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,142 INFO [ ome.services.util.ServiceHandler] (.Server-13) Rslt: false
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,143 INFO [ omero.cmd.SessionI] (.Server-13) Added servant to adapter: f740349b-7e94-42fe-a99e-dbd75fb00332/1ea1baca-a804-4966-9880-be092746d1b4omero.api.MetadataStore(omero.api._Meta
dataStoreTie@eb02f950)
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO [ omero.cmd.SessionI] (l.Server-4) Unregistered servant:f740349b-7e94-42fe-a99e-dbd75fb00332/IHandle1b464420-82e0-45d3-970b-0c4c344cc770(omero.cmd._HandleTie@30297944)
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Closing...
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Handle closed before finished! State=READY
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Cancelling...
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Cancelled
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) notify cancelled: null/omero.cmd.Status@4240ce1e
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) notify cancelled: null/omero.cmd.Status@4240ce1e
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,290 INFO [ ome.formats.OMEROMetadataStoreClient] (2-thread-4) Pinging session every 300s.
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,306 INFO [ ome.services.util.ServiceHandler] (l.Server-3) Meth: interface ome.api.StatefulServiceInterface.close
Dom saw the same error for idr0138 NGFF import at https://idr-redmine.openmicroscopy.org/issues/266#note-20 Solved by not importing in parallel.
We now have s3-support in ZarrReader, using a .zarr.bfoptions
file as a sibling beside the .zarr
directory (symlink) that contains the alt_store
in addition to other flags that tell ZarrReader to optimise performance, e.g. making assumptions that all images are same dimensions etc.
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://path/to/data.zarr`
Test import on pilot-idrngff
which has recent ZarrReader AND Bio-Formats jars updated from daily build:
bash-5.1$ ls -alh /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_b45.jar
-rw-r--r--. 1 omero-server omero-server 26M Apr 22 12:06 /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_b45.jar
bash-5.1$ ls -alh /opt/omero/server/OMERO.server/lib/server/ | grep SNAPSHOT
-rw-r--r--. 1 omero-server omero-server 143K Apr 25 15:57 formats-api-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 880K Apr 25 15:57 formats-bsd-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.5M Apr 25 15:57 formats-gpl-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 14K Apr 25 15:57 metakit-5.3.8-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 7.8K Apr 25 15:57 mipav-stubs-6.0.3-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 90K Apr 25 15:57 ome-codecs-1.0.4-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 139K Apr 25 15:57 ome-common-6.0.23-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.0M Apr 25 15:57 ome-jai-0.1.5-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 90K Apr 25 15:57 ome-mdbtools-5.3.4-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 928K Apr 25 15:57 ome-poi-5.3.10-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 639K Apr 25 15:57 ome-xml-6.3.7-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 615K Apr 25 15:57 specification-6.3.7-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.3M Apr 25 15:57 turbojpeg-8.0.0-SNAPSHOT.jar
Want to install aws..
$ sudo dnf install awscli
...
Installed:
awscli-1.23.10-3.el9.noarch groff-1.22.4-10.el9.x86_64 paper-2.3-6.el9.x86_64 perl-File-Copy-2.34-480.el9.noarch perl-IPC-Run3-0.048-23.el9.noarch perl-locale-1.09-480.el9.noarch
psutils-2.07-1.el9.noarch python3-botocore-1.25.10-1.el9.noarch python3-colorama-0.4.4-12.el9.noarch python3-docutils-0.16-6.el9.noarch python3-jmespath-0.9.4-11.el9.noarch python3-pyasn1-0.4.8-6.el9.noarch
python3-rsa-4.9-2.el9.noarch python3-s3transfer-0.5.2-2.el9.noarch
Complete!
$ which aws
/usr/bin/aws
Download metadata-only plates as omero-server...
sudo -u omero-server -s
cd
mkdir idr0125 && cd idr0125
vi idr0125_plates.csv. # added first 5 plates from list above
for zarr in $(cat idr0125_plates.csv);
do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done
Import...
omero login
export OMERODIR=/opt/omero/server/OMERO.server
for zarr in $(cat idr0125_plates.csv);
do omero import --depth=100 --transfer=ln_s --skip=all idr0125/$zarr --file /tmp/idr0125_$zarr.log --errs /tmp/idr0125_$zarr.err;
done
log... First plate...
...
2024-04-30 14:34:27,973 13708373 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320647
==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 3:48:16.912
Let's try to create a new bfoptions file for each Fileset as at https://github.com/IDR/idr-metadata/issues/684#issuecomment-1886332782 with the appropriate data
For first Plate, with Fileset paths like: demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/I/21/5/0/.zarray
We want to generate .bfoptions like this for first Plate
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
First Plate Fileset ID is 6320647
. First Image is 15154987
Need to check which file is referred to in Pixels Table:
idr=> select path, name from Pixels where image=15154987;
path | name
--------------------------------------------------------------------------------------------------------------------------+------------------
demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME | METADATA.ome.xml
(1 row)
As omero-server on pilot-idrngff
For ZarrReader, the bfoptions
file is a sibling of the .zarr
directory:
export bfoptionspath="/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions"
export fsid=6320647
echo "omezarr.list_pixels=false" > $bfoptionspath;
echo "omezarr.quick_read=true" >> $bfoptionspath;
echo "omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr" >> $bfoptionspath;
psql -U omero -d idr -h $DBHOST -f addbfoptions.sql
CREATE FUNCTION
psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions', '$SECRET')"
add_origfile_to_fileset
-------------------------
61623686
(1 row)
rm -f /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo
Try to view image... 10:40...
Saving memo was very fast (300 ms!?):
(venv3) bash-5.1$ grep -B 3 "10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -B 3 "saved memo"
2024-05-01 09:40:32,659 INFO [ omero.cmd.SessionI] (l.Server-5) Removed servant from adapter: be265e12-20c1-45fb-bcdf-8695e703ddf5omero.api.IScript
2024-05-01 09:40:37,416 DEBUG [ loci.formats.Memoizer] (l.Server-7) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo6657480071468204245
2024-05-01 09:40:37,416 DEBUG [ loci.formats.Memoizer] (l.Server-7) start[1714556437115] time[300] tag[loci.formats.Memoizer.saveMemo]
2024-05-01 09:40:37,416 DEBUG [ loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6406530 bytes)
Images are viewable! - approx 6 seconds to render a plane (we only have a single resolution level - added comment at https://github.com/broadinstitute/lincs-cell-painting/issues/54#issuecomment-2088292778)
Save To All
resulted in many thumbnails being updated, but at ~6 seconds each this eventually timed-out, so we only have thumbnails for part of the plate:
To run the above bfoptions generation for "all" plates...
Need to generate csv with:
6320647
demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/
SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Used https://github.com/IDR/idr0125-way-cellpainting/blob/0a59dab08898e0deb4e75d5b6a80f36e94ac9315/scripts/get_filesets_to_csv.py to generate
idr0125_filesets.csv
6320647,SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/
6320648,SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-3/2024-04/30/14-34-41.406/
6320649,SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-11/2024-04/30/18-29-40.187/
6320650,SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-8/2024-04/30/22-34-25.404/
6320651,SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-0/2024-05/01/02-28-45.532/
psql -U omero -d idr -h $DBHOST -f addbfoptions.sql
CREATE FUNCTION
export s3url="https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr"
for r in $(cat idr0125_filesets.csv); do
fsid=$(echo $r | cut -d',' -f1)
zarrname=$(echo $r | cut -d',' -f2)
prefix=$(echo $r | cut -d',' -f3 | sed 's/.$//')
bfoptionspath="/data/OMERO/ManagedRepository/$prefix/$zarrname.bfoptions"
echo $bfoptionspath
echo "omezarr.list_pixels=false" > $bfoptionspath
echo "omezarr.quick_read=true" >> $bfoptionspath
echo "omezarr.alt_store=$s3url/$zarrname" >> $bfoptionspath
psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, '$zarrname.bfoptions', '$SECRET')"
rm /data/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/$zarrname/OME/.METADATA.ome.xml.bfmemo
done
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2024-04/30/14-34-41.406/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions
add_origfile_to_fileset
-------------------------
61623688
(1 row)
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2024-04/30/18-29-40.187/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr.bfoptions
add_origfile_to_fileset
-------------------------
61623689
(1 row)
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2024-04/30/22-34-25.404/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr.bfoptions
add_origfile_to_fileset
-------------------------
61623690
(1 row)
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2024-05/01/02-28-45.532/SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr.bfoptions
add_origfile_to_fileset
-------------------------
61623691
(1 row)
Plate images are viewable 👍
Copy rendering settings from image viewed above (settings applied in webclient) to the 5 plates imported to date...
Run in a screen...
for i in {10551..10555}; do
echo "$i"
omero --debug=2 render copy Image:15154987 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ tail -f /tmp/idr0125_copyplate_10551.err
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Image:15154988 got thumbnail in 8.03s
Image:15154989 got thumbnail in 54.77s
Image:15154990 got thumbnail in 62.49s
Image:15154991 got thumbnail in 34.76s
Image:15154992 got thumbnail in 49.56s
...
Rendering of thumbnails took ~16 hours for the first plate - approx 16 secs per image on average.
First plate took ~16 hours, 2nd plate took 24 hours, then last 3 plates took about 12 hours each. This is comparable to above (before s3 support in ZarrReader) e.g. 13 hours at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1785591507 and after.
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ ls -alh /tmp/idr0125_copyplate_*
-rw-r--r--. 1 omero-server omero-server 130K May 2 05:57 /tmp/idr0125_copyplate_10551.err
-rw-r--r--. 1 omero-server omero-server 2.9K May 2 05:57 /tmp/idr0125_copyplate_10551.log
-rw-r--r--. 1 omero-server omero-server 131K May 3 06:12 /tmp/idr0125_copyplate_10552.err
-rw-r--r--. 1 omero-server omero-server 2.9K May 3 06:12 /tmp/idr0125_copyplate_10552.log
-rw-r--r--. 1 omero-server omero-server 130K May 3 18:06 /tmp/idr0125_copyplate_10553.err
-rw-r--r--. 1 omero-server omero-server 2.9K May 3 18:06 /tmp/idr0125_copyplate_10553.log
-rw-r--r--. 1 omero-server omero-server 130K May 4 07:27 /tmp/idr0125_copyplate_10554.err
-rw-r--r--. 1 omero-server omero-server 2.9K May 4 07:27 /tmp/idr0125_copyplate_10554.log
-rw-r--r--. 1 omero-server omero-server 129K May 4 16:27 /tmp/idr0125_copyplate_10555.err
-rw-r--r--. 1 omero-server omero-server 2.9K May 4 16:27 /tmp/idr0125_copyplate_10555.log
Checking the .log
files, I see that all 3456 thumbnails are generated for all plates except first plate (1 less since the original image is there) and for the 2nd plate:
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ cat /tmp/idr0125_copyplate_10552.log
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 98 images.
Rendering settings successfully copied to 93 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 100 images.
Rendering settings successfully copied to 56 images.
This is missing 2 + 7 images: Browsing the data I can see missing thumbnails for Wells:
These Images all have Channels named e.g. 0, 1, 2, 3, 4
in webclient, imgData
JSON has "label": "0",
.
This would explain why the copy of rendering settings failed above.
However, the original NGFF data has:
"label": "Channel 0"
E.g. https://ome.github.io/ome-ngff-validator/?source=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/O/1/7/
The images appear to have "black" 0-array chunks (rather than missing chunks) of 9.2 kb each:
We can manually fix this small number of images by copy, paste & save of rendering settings in webclient.
Sample plate at https://ome.github.io/ome-ngff-validator/?source=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/ has downsampled resolutions...
Need to download and re-import metadata only plate etc...
aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
omero import --depth=100 --transfer=ln_s --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125_withdownscale.log --errs /tmp/idr0125_withdownscale.err
...
2024-05-17 16:44:19,239 17201442 [l.Client-5] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61634458
2024-05-17 16:44:21,393 17203596 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/withdownscale/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320652
==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 4:46:17.439
Create bfoptions...
vi /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-05/17/11-58-05.161/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions', '$SECRET')"
rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository//demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-05/17/11-58-05.161/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo
omero --debug=2 render copy Image:15172267 Plate:10556 > /tmp/idr0125_copyplate_20240519.log 2>/tmp/idr0125_copyplate_20240519.err
Looks faster!
Image:15172272 got thumbnail in 2.05s
Image:15172273 got thumbnail in 1.66s
Image:15172274 got thumbnail in 1.61s
Image:15172275 got thumbnail in 1.53s
Image:15172276 got thumbnail in 1.54s
Image:15172277 got thumbnail in 1.63s
Image:15172278 got thumbnail in 1.42s
...
EDIT: Done after approx 1.5 hours.
On pilot-idrngff
, setup goofys mount of cellpainting-gallery
so we can symlink to it after import. We won't have any other local or mounted copy of the data, so this will be essential for memo file generation (ZarrReader can use the public s3 link in bfoptions for subsequent pixel access, but this is seems not sufficient for future memo file generation).
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo mkdir /cellpainting-gallery
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo goofys -o allow_other cellpainting-gallery /cellpainting-gallery/
Create idr0125_plates.csv
from above https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1754891150
bash-5.1$ pwd
/opt/omero/server/idr0125
bash-5.1$ wc idr0125_plates.csv
136 136 7344 idr0125_plates.csv
In a screen...
for zarr in $(cat idr0125_plates.csv);
do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done
~6 hours later:
for zarr in $(cat idr0125_plates.csv);
do omero import --depth=100 --transfer=ln_s --skip=all idr0125/$zarr --file /tmp/idr0125_$zarr.log --errs /tmp/idr0125_$zarr.err;
done
First import log
...
2024-06-05 01:30:06,683 15009142 [3-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zgroup
2024-06-05 01:45:27,401 15929860 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2024-06-05 01:45:29,347 15931806 [2-thread-1] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61659846
2024-06-05 02:22:04,606 18127065 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 61659846
2024-06-05 02:41:29,988 19292447 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 61659846
2024-06-05 02:41:30,049 19292508 [l.Client-5] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 61659846
2024-06-05 02:41:30,085 19292544 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 61659846
2024-06-05 02:41:30,114 19292573 [l.Client-5] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 61659846
2024-06-05 02:41:36,564 19299023 [l.Client-4] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320653
==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 5:11:36.992
Takes approx 5 hours instead of under 4 hours, probably due to extra resolution level
For first 5 Plates, put these into a Screen:3502 so we can update symlinks... Using branch https://github.com/IDR/idr-utils/pull/54
cd /uod/idr/metadata/idr-utils/
$ sudo -Es git fetch will
$ sudo -Es git checkout will/managed_repo_symlinks
As omero-server
screen -S idr0125_symlinks
source ~/venv3/bin/activate
cd /data/idr-metadata/idr-utils
omero login
# check mount - should list plates
ls /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr
$ python scripts/managed_repo_symlinks.py Screen:3502 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
Took less than 10 mins for 5 plates. Images are now viewable with chunks.
Rendering times (without bfoptions s3) in Preview panel is about 0.5 secs or less for low-resolution image plane (3 channels) and 2-3 seconds for full-resolution tiles.
Try setting rendering settings and thumbnails without bfoptions, for first 5 plates imported above... 16:49...
for i in {10557..10561}; do
echo "$i"
omero --debug=2 render copy Image:15175723 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done
Logs show 2-3 secs per Thumbnail...
Image:15175724 got thumbnail in 3.75s
Image:15175725 got thumbnail in 3.11s
Image:15175726 got thumbnail in 3.65s
Image:15175727 got thumbnail in 3.11s
Image:15175728 got thumbnail in 3.25s
Image:15175729 got thumbnail in 3.21s
Image:15175730 got thumbnail in 2.60s
Image:15175731 got thumbnail in 2.87s
Image:15175732 got thumbnail in 2.86s
Image:15175733 got thumbnail in 0.31s
Image:15175734 got thumbnail in 2.73s
Image:15175735 got thumbnail in 3.48s
Image:15175736 got thumbnail in 3.20s
Image:15175737 got thumbnail in 3.39s
Image:15175738 got thumbnail in 2.99s
Image:15175739 got thumbnail in 2.88s
...and 3 hours per plate:
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ ls -alh /tmp | grep copyplate
-rw-r--r--. 1 omero-server omero-server 129K Jun 6 18:46 idr0125_copyplate_10557.err
-rw-r--r--. 1 omero-server omero-server 2.9K Jun 6 18:46 idr0125_copyplate_10557.log
-rw-r--r--. 1 omero-server omero-server 129K Jun 6 21:46 idr0125_copyplate_10558.err
-rw-r--r--. 1 omero-server omero-server 2.9K Jun 6 21:46 idr0125_copyplate_10558.log
-rw-r--r--. 1 omero-server omero-server 129K Jun 7 00:44 idr0125_copyplate_10559.err
-rw-r--r--. 1 omero-server omero-server 2.9K Jun 7 00:44 idr0125_copyplate_10559.log
-rw-r--r--. 1 omero-server omero-server 129K Jun 7 03:49 idr0125_copyplate_10560.err
-rw-r--r--. 1 omero-server omero-server 2.9K Jun 7 03:49 idr0125_copyplate_10560.log
-rw-r--r--. 1 omero-server omero-server 129K Jun 7 06:53 idr0125_copyplate_10561.err
-rw-r--r--. 1 omero-server omero-server 2.9K Jun 7 06:53 idr0125_copyplate_10561.log
Import still running from 3 weeks ago https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2147163019
Now we have 90 newly-imported Plates (in addition to the 5 above). Put them in a Screen in webclient and update symlinks as in previous comment:
python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
...
Fileset: 6320745 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-44/2024-06/24/23-22-39.294/
Render Image 15493675
fs_contents ['SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-44/2024-06/24/23-22-39.294/SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr
Fileset: 6320746 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-41/2024-06/25/04-21-54.420/
Render Image 15497131
fs_contents ['SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-41/2024-06/25/04-21-54.420/SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr
Fileset: 6320747 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-40/2024-06/25/09-28-38.106/
Render Image 15500587
fs_contents ['SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-40/2024-06/25/09-28-38.106/SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr
Previously we don't need to add the bfoptions
to the Fileset (as we did for idr0009 and tested above) since ZarrReader will check for the .bfoptions
file regardless (as it does for all other NGFF data)!
Test on one Plate... First of the 5 imported above: http://localhost:1080/webclient/?show=plate-10557
demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Need to create .bfoptions at
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
Delete memo...
rm -f /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo
Taking a while to regenerate (viewing in webclient)...
Eventually done - logs claim 414 milli-secs - although this completed at 17:06
BST (16:06 GMT) and was started some time ago!
grep -B 3 "21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -B 3 "saved memo"
2024-06-26 16:06:45,194 INFO [ omero.cmd.SessionI] (.Server-43) Removed servant from adapter: eaaf2e19-0308-470a-a645-13705799f767omero.api.IQuery
2024-06-26 16:06:48,659 DEBUG [ loci.formats.Memoizer] (.Server-37) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo6092143311287441869
2024-06-26 16:06:48,659 DEBUG [ loci.formats.Memoizer] (.Server-37) start[1719418008245] time[414] tag[loci.formats.Memoizer.saveMemo]
2024-06-26 16:06:48,659 DEBUG [ loci.formats.Memoizer] (.Server-37) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996260 bytes)
Previously, when bfoptions file was added to the Fileset, memo file regeneration was very fast (300 ms)! https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2088306464 So let's try that with another Plate - 2nd of the bunch of 5... http://localhost:1080/webclient/?show=plate-10558
demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr
Create bfoptions at
/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions
with
omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr
$ psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset(6320654, 'SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions', '$SECRET')"
add_origfile_to_fileset
-------------------------
64161893
(1 row)
Delete memo...
rm /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo
View in webclient.... again, memo file generation is not quick....
Copy rendering settings from first Image to the rest of first Plate...
omero --debug=2 render copy Image:15175723 Plate:10557 > /tmp/idr0125_copyplate_10557.log 2>/tmp/idr0125_PR59_copyplate_10557.err
Seems that not all 90 plates in Screen above have had symlinks created correctly at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2189178399 The 3 Plates listed there are OK (can see chunks loading in webclient) but others not working.
Try re-running:
$ ssh -A pilot-idrngff -L 1080:localhost:80
$ sudo -u omero-server -s
$ screen -r idr0125_symlinks
$ source /opt/omero/server/venv3/bin/activate
$ cd /uod/idr/metadata/idr-utils/
$ python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
fileset_dirs {}
Fileset: 6320658 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2024-06/05/22-56-44.206/
Render Image 15193003
fs_contents ['SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2024-06/05/22-56-44.206/SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr
Fileset: 6320659 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/
Render Image 15196459
fs_contents ['SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr
Fileset: 6320660 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/06/08-42-04.670/
Render Image 15199915
fs_contents ['SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/06/08-42-04.670/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr
Fileset: 6320661 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-37/2024-06/06/13-32-34.540/
Render Image 15203371
fs_contents ['SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-37/2024-06/06/13-32-34.540/SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr
Fileset: 6320662 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2024-06/06/18-46-57.670/
Render Image 15206827...
While browsing webclient while still running, got DatabaseBusyException
Re-ran
python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240806.log
Looks good this time - All 94 Plates completed... in ~3 hours Script started at 6.40 BST, completed 9:30 (based on time of last link 8:30 GMT)
bash-5.1$ grep "Render Image" /tmp/idr0125_symlinks_20240806.log | wc
94 282 2068
bash-5.1$ ls -lh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-38/2024-06/26/06-17-38.755/
total 0
lrwxrwxrwx. 1 omero-server omero-server 142 Aug 6 08:30 SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr
Repeat with remaining 37 plates in temp Screen:3504 - add to same log...
python scripts/managed_repo_symlinks.py Screen:3504 cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240806.log
EDIT (7th Aug) - Mistake in previous command - missed the leading / Running again (much faster since Images are rendered already)
$ python scripts/managed_repo_symlinks.py Screen:3504 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240807.log
(venv3) bash-5.1$ grep "Link from" /tmp/idr0125_symlinks_20240807.log | wc
37 185 11248
Trying to view images in webclient to trigger memo file regeneration:
Taking a very long time, with occasional DatabaseBusy exceptions -> restart server...
Checking logs for memo saving times for one plate give 4 results, all starting within a couple of seconds and taking 5 hours! I wonder if clicking in webclient can trigger multiple memo generations in parallel? Probably better to use script.
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ grep -A 3 "SQ00014818__2016-05-23T10_30_25" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 3 "saved memo"
2024-08-08 03:32:16,840 DEBUG [ loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:16,840 DEBUG [ loci.formats.Memoizer] (l.Server-3) start[1723069481615] time[18455225] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:16,840 INFO [ ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 1575
2024-08-08 03:32:16,847 INFO [ ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml
--
2024-08-08 03:32:16,909 DEBUG [ loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:16,909 DEBUG [ loci.formats.Memoizer] (l.Server-1) start[1723069491280] time[18445628] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:16,909 INFO [ ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 0
2024-08-08 03:32:16,910 INFO [ org.perf4j.TimingLogger] (l.Server-1) start[1723069491275] time[18445635] tag[omero.call.success.ome.services.RenderingBean$12.doWork]
--
2024-08-08 03:32:17,098 DEBUG [ loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:17,099 DEBUG [ loci.formats.Memoizer] (l.Server-4) start[1723069481453] time[18455645] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:17,099 INFO [ ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 648
2024-08-08 03:32:17,145 INFO [ome.services.sessions.state.SessionCache] (2-thread-3) Destroying session 00807f1d-2eeb-474f-8c81-2d0cdad62308 due to : Timeout
--
2024-08-08 03:32:17,160 DEBUG [ loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:17,160 DEBUG [ loci.formats.Memoizer] (l.Server-5) start[1723069491451] time[18445708] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:17,160 INFO [ ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 1
2024-08-08 03:32:17,189 INFO [ ome.services.util.ServiceHandler] (2-thread-3) Executor.doWork -- ome.services.sessions.SessionManagerImpl.reload[d03aac19-dbb9-477f-8983-298029bc6a0a]
Memo file generation: As omero-server user
omero hql --limit -1 --ids-only --style csv 'select MIN(field.image.id) FROM WellSample AS field where field.well.plate.id in (select link.child.id from ScreenPlateLink as link where link.parent.id=3503) GROUP BY field.well.plate' > idr0125_plates.txt
Remove first row, then
cut -d ',' -f2 idr0125_plates.txt | sed -e 's/^/Image:/' > idr0125_plate_ids.txt
screen -dmS cache parallel --eta -a idr0125_plate_ids.txt --results /tmp/memo_idr0125_20240808/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'
Memo generation took about an hour for each Plate (first 3 were already done):
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ grep ok /tmp/memo_idr0125_20240808/1/**/**
/tmp/memo_idr0125_20240808/1/Image:15193003/stdout:ok: Pixels:15193003 Image:15193003 2.009263277053833
/tmp/memo_idr0125_20240808/1/Image:15196459/stdout:ok: Pixels:15196459 Image:15196459 2.880405902862549
/tmp/memo_idr0125_20240808/1/Image:15199915/stdout:ok: Pixels:15199915 Image:15199915 2.134599447250366
/tmp/memo_idr0125_20240808/1/Image:15203371/stdout:ok: Pixels:15203371 Image:15203371 3689.8124442100525
/tmp/memo_idr0125_20240808/1/Image:15206827/stdout:ok: Pixels:15206827 Image:15206827 3754.8339743614197
/tmp/memo_idr0125_20240808/1/Image:15210283/stdout:ok: Pixels:15210283 Image:15210283 3674.200692176819
...
and the last one completed about 7am on the 9th (today). Took approx 13 hours. 👍
Copy rendering settings to 131 plates... Instead of for loop (above)...
for i in {10562..10692}; do
echo "$i"
omero --debug=2 render copy Image:15175723 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done
Let's try to use parallel... First make list of plates (all sequential IDs):
for i in {10562..10692}; do
echo "Plate:$i" >> idr0125_ids_plates.txt
done
Then... using demo login...
export $PASS=secret
screen -dmS cache parallel --eta -a idr0125_ids_plates.txt --results /tmp/copyplate_idr0125_20240809/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u demo -w $PASS copy Image:15175723'
This issue has been mentioned on Image.sc Forum. There might be relevant details there:
for i in {10562..10692}; do grep "100 images" /tmp/copyplate_idr0125_20240809/1/Plate:$i/stdout | wc done
Not 34 in the following Plates:
10568 (33) - SQ00015043 P9 Field 1 10605 (33) - SQ00015120 P1 Field 6 10633 (33) - SQ00015148 P24 Field 6 10657 (33) - SQ00015173 O19 Field 2 10659 (33) - SQ00015195 B23 Field 5 10661 (33) - SQ00015197 P1 Field 1 10662 (32) - SQ00015198 P1 Fields 2-8, P21 Field 2 10671 (33) - SQ00015207 P1 (missing Well), O1 Fields 1, 3-9, 10672 (33) - SQ00015208 B22 Field 5
Annotations...
(base) [wmoore@pilot-idrngff-omeroreadwrite metadata]$ sudo -Es git clone git@github.com:IDR/idr0125-way-cellpainting.git
omero-server
$ omero metadata populate --report --batch 1000 --file screenA/idr0125-screenA-annotation.csv Screen:3503
On idr0125-pilot, going to test import of METADATA.ome.xml and then use mkngff to replace the fileset with full NGFF plate mounted on s3.
Running mkngff to replace OME.METADATA.ome.xml single-file Fileset with NGFF plate...