IDR / idr0125-way-cellpainting

0 stars 1 forks source link

idr0125 NGFF import #4

Open will-moore opened 11 months ago

will-moore commented 11 months ago

On idr0125-pilot, going to test import of METADATA.ome.xml and then use mkngff to replace the fileset with full NGFF plate mounted on s3.

sudo cp idr0125/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr/OME/METADATA.ome.xml /opt/omero/server/
import as omero-server user...

omero import --transfer=ln_s --skip=all METADATA.ome.xml --file /tmp/idr0125_xml.log  --errs /tmp/idr0125_xml.err
$ tail -f /tmp/idr0125_xml.err
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-09-04 16:00:07,716 15221      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-09-04 16:00:07,994 15499      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-09-04 16:00:31,170 38675      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-09-04 16:00:31,188 38693      [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /opt/omero/server/idr0125/METADATA.ome.xml...
2023-09-04 16:00:31,412 38917      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,219 39724      [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/idr0125/METADATA.ome.xml
2023-09-04 16:00:32,391 39896      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-09-04 16:00:32,673 40178      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 52201342

Other imported objects:
Fileset:5287519

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 3447 images imported, 0 errors in 0:59:35.161

Running mkngff to replace OME.METADATA.ome.xml single-file Fileset with NGFF plate...

$ omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET 5287519 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr > 5287519.sql

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04 // 16-00-19.149 for fileset 5287519
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-141/2023-09/04/16-00-19.149_mkngff/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015207__2016-04-24T05_39_26-Measurement1.ome.zarr

$ psql -U omero -d idr -h $DBHOST -f 5287519.sql 
UPDATE 3447
BEGIN
 mkngff_fileset 
----------------
        5287569
(1 row)

COMMIT

will-moore commented 9 months ago

Import done. Need to update symlinks for 11151-11219. Since these were imported with ZarrReader.jar from performance PR, we have extra /data/ngff in path, which needs correct commit of managed_repo_symlinks PR:

$ sudo -Es git checkout 2888b21c2751421ecf24a
...
2888b21... TEMP add /ngff/idr0125 to templatePrefix for each fileset

As omero-server... 10:38...

for i in {11151..11219}; do
    echo "$i"
    python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
done

EDIT: 11:00... leave running..

Fileset: 5288583 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-39/2023-11/08/14-55-46.675/ngff/idr0125
Render Image 16296891
fs_contents ['SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-39/2023-11/08/14-55-46.675/ngff/idr0125/SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015151__2016-06-09T05_07_35-Measurement1.ome.zarr
11164
fileset_dirs {}

Fileset: 5288584 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-50/2023-11/08/17-29-10.806/ngff/idr0125
Render Image 16298674

Tried browsing webclient in preparation for thumbnailing... Got DatabaseBusyException in webclient. symlinking got to...

    preview_image(conn, fileset_id, args)
  File "idr-utils/scripts/managed_repo_symlinks.py", line 62, in preview_image
    image.renderJpeg()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7949, in wrapped
    except ctx.ignoreExceptions:
TypeError: catching classes that do not inherit from BaseException is not allowed
11177
fileset_dirs {}

Can't login - need $ sudo service omero-server restart...

12:41...

for i in {11176..11219}; do     echo "$i";     python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report; done

...
11219
fileset_dirs {}
Fileset: 5288639 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2023-11/14/10-23-13.402/ngff/idr0125
Render Image 16491466
fs_contents ['SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2023-11/14/10-23-13.402/ngff/idr0125/SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015233__2016-05-12T05_47_03-Measurement1.ome.zarr

completed (last link created at) 13:53. 70 mins for 43 symlinks

will-moore commented 9 months ago

Browsing webclient to check links before running render...

        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328)
        ... 57 common frames omitted
2023-11-16 16:13:48,736 INFO  [                 org.perf4j.TimingLogger] (.Server-38) start[1700151134730] time[94006] tag[omero.call.exception]
2023-11-16 16:13:48,737 WARN  [        ome.services.util.ServiceHandler] (.Server-38) CannotCreateTransactionException thrown.

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBus
yException: Cannot acquire connection

This is without running anything else just now...

Restart server again...

Browsing a different plate on webclient - again...

Ice.UnknownException: exception ::Ice::UnknownException
{
    unknown = ome.conditions.DatabaseBusyException: cannot create transaction
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)

will-moore commented 9 months ago

After leaving the server for a few days...

for i in {11053..11058}; do echo "$i"; python idr-utils/scripts/managed_repo_symlinks.py Plate:$i /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report; done

Tried again viewing an Image from the 3rd Plate: Plate: 11053, SQ00014814__2016-05-23T17_24_56-Measurement1. I don't have a record of creating symlinks for that Plate, but the symlink and chunks seem to be there:

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/
total 0
drwxrwsr-x. 2 omero-server omero-server  67 Oct 31 09:57 .
drwxrwsr-x. 4 omero-server omero-server  94 Oct 30 17:11 ..
lrwxrwxrwx. 1 omero-server omero-server 142 Oct 31 09:57 SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/1/0/0/0/0/0/0
total 4.3M
drwxr-xr-x. 2 root root 4.0K Dec  7  2022 .
drwxr-xr-x. 2 root root 4.0K Dec  7  2022 ..
-rw-r--r--. 1 root root 2.0M Dec  7  2022 0
-rw-r--r--. 1 root root 2.1M Dec  7  2022 1
-rw-r--r--. 1 root root 239K Dec  7  2022 2

However, images from the Plate are not viewable. Tried browsing the next 3 plates

    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource dfce81f1-9a82-4a7c-b00e-4452ee77a8ba still empty after 30s wait time
    at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:319)
    at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181)
    at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150)
    at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:328)
    ... 25 common frames omitted
2023-11-20 12:41:30,742 ERROR [     o.s.blitz.fire.PermissionsVerifierI] (.Server-34) Exception thrown while checking password for:public
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is ome.conditions.DatabaseBusyException: Cannot acquire connection
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy85.doWork(Unknown Source)
    at ome.services.util.Executor$Impl.executeSql(Executor.java:549)
    at ome.services.blitz.fire.Ring.checkPassword(Ring.java:270)
    at ome.services.blitz.fire.PermissionsVerifierI.checkPermissions(PermissionsVerifierI.java:97)
    at Glacier2._PermissionsVerifierDisp.___checkPermissions(_PermissionsVerifierDisp.java:114)
    at Glacier2._PermissionsVerifierDisp.__dispatch(_PermissionsVerifierDisp.java:149)
    at IceInternal.Incoming.invoke(Incoming.java:221)
    at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
    at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
    at Ice.ConnectionI.message(ConnectionI.java:1056)
    at IceInternal.ThreadPool.run(ThreadPool.java:395)
    at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
    at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: ome.conditions.DatabaseBusyException: Cannot acquire connection
    at ome.services.db.SelfCorrectingDataSource.callWithRetries(SelfCorrectingDataSource.java:113)
    at ome.services.db.SelfCorrectingDataSource.getConnection(SelfCorrectingDataSource.java:60)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
    ... 20 common frames omitted
2023-11-20 12:41:31,872 INFO  [ ome.services.blitz.fire.SessionManagerI] (.Server-34) Found session locally: 416569f8-07bc-44df-afdd-bea2d004cfa1
2023-11-20 12:41:31,873 INFO  [ ome.services.blitz.fire.SessionManagerI] (.Server-34) Rejoining session ServiceFactoryI(session-ba637349-1805-4f29-a971-4e26340c25c1/416569f8-07bc-44df-afdd-bea2d004cfa1) (agent=OMERO.web)
2023-11-20 12:41:31,878 INFO  [                      omero.cmd.SessionI] (.Server-35) Added servant to adapter: 416569f8-07bc-44df-afdd-bea2d004cfa1/ba637349-1805-4f29-a971-4e26340c25c1omero.api.IConfig(omero.api._IConfigTie@a7b89013)

will-moore commented 9 months ago

Also still seeing ZarrReader logs...

...
2023-11-20 12:54:19,871 ERROR [              loci.formats.FormatHandler] (l.Server-0) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
2023-11-20 12:54:19,871 ERROR [              loci.formats.FormatHandler] (l.Server-7) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
2023-11-20 12:54:19,871 ERROR [              loci.formats.FormatHandler] (l.Server-1) Adding to the used files list: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-10/30/16-51-43.415/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr/A/15/2/0/.zarray
...

will-moore commented 8 months ago

Returning to idr0125-pilot again (since updating the ZarrReader to test idr0015 NGFF data) it looks like all the idr0125 data imported above is not viewable, possibly due to memo file regeneration.

Since we now have ZarrReader-0.4.0 released now with the NGFF performance improvements, lets install that and start a fresh import of all data...

wget https://artifacts.openmicroscopy.org/artifactory/ome.releases/ome/OMEZarrReader/0.4.0/OMEZarrReader-0.4.0.jar
...update, retstart etc...

sudo -Es git checkout -- screenA/idr0125-screenA-plates.tsv

Renamed previous idr0125 Screen in webclient, so fresh one should get created... bulk.yml still commented out the clientpath check

omero import --depth=100 --transfer=ln_s --bulk screenA/idr0125-screenA-bulk.yml --file /tmp/idr0125_20231214.log  --errs /tmp/idr0125_20231214.err

will-moore commented 8 months ago

Import failed...

2023-12-14 10:19:52,383 687        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.2
2023-12-14 10:19:52,425 729        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.12.0 revision: 9d2f3cc080ff771569ca08fb7b3fe3558b8396f2 date: 14 February 2023
2023-12-14 10:19:52,511 815        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-12-14 10:19:52,516 820        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2023-12-14 10:19:52,516 820        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping thumbnails creation
2023-12-14 10:19:52,516 820        [      main] INFO   formats.importer.cli.CommandLineImporter - Skipping minimum/maximum computation
2023-12-14 10:19:52,516 820        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2023-12-14 10:19:52,516 820        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2023-12-14 10:19:52,517 821        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting output format: yaml
2023-12-14 10:19:52,517 821        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting parallel upload: 10
2023-12-14 10:19:52,517 821        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting parallel fileset: 10
2023-12-14 10:19:52,517 821        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-12-14 10:19:55,317 3621       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-12-14 10:23:27,435 215739     [      main] INFO      ome.formats.importer.ImportCandidates - 11157 file(s) parsed into 1 group(s) with 1 call(s) to setId in 190658ms. (212117ms total) [0 unknowns]
2023-12-14 10:23:29,035 217339     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2023-12-14 10:23:31,174 219478     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-12-14 10:23:31,991 220295     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-12-14 10:23:32,003 220307     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-12-14 10:23:32,004 220308     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-12-14 10:23:32,004 220308     [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-12-14 10:23:32,004 220308     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-12-14 10:23:32,004 220308     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-12-14 10:23:32,004 220308     [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-12-14 10:23:32,805 221109     [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Screen:name:idr0125-way-cellpainting/screenA
2023-12-14 10:23:33,180 221484     [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Screen:3501
2023-12-14 10:23:33,186 221490     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2023-12-14 11:27:50,970 4079274    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2023-12-14 11:27:51,022 4079326    [3-thread-3] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/.zattrs...
2023-12-14 11:27:51,022 4079326    [3-thread-2] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/0/.zarray...
...
2023-12-14 11:36:56,558 4624862    [3-thread-9] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/8/.zattrs
2023-12-14 11:36:56,566 4624870    [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zgroup
2023-12-14 11:36:56,927 4625231    [3-thread-4] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/.zgroup
2023-12-14 11:46:15,888 5184192    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-12-14 11:46:16,286 5184590    [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
Ice.UnmarshalOutOfBoundsException: null
        at IceInternal.BasicStream.readSize(BasicStream.java:632)
        at IceInternal.BasicStream$EncapsDecoder11.readObject(BasicStream.java:3543)
        at IceInternal.BasicStream.readObject(BasicStream.java:2262)
...
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-12-14 11:46:16,287 5184591    [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Continuing after error

==> Summary
11157 files uploaded, 0 filesets created, 0 images imported, 0 errors in 1:22:43.987

Looking at Blitz logs around then doesn't show much...

grep "2023-12-14 11:46:16" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.* | less

/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,142 INFO  [                 org.perf4j.TimingLogger] (.Server-13) start[1702554376139] time[3] tag[omero.call.success.ome.services.blitz.impl.ServiceFactoryI$3.doWork]
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,142 INFO  [        ome.services.util.ServiceHandler] (.Server-13)  Rslt:       false
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,143 INFO  [                      omero.cmd.SessionI] (.Server-13) Added servant to adapter: f740349b-7e94-42fe-a99e-dbd75fb00332/1ea1baca-a804-4966-9880-be092746d1b4omero.api.MetadataStore(omero.api._Meta
dataStoreTie@eb02f950)
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO  [                      omero.cmd.SessionI] (l.Server-4) Unregistered servant:f740349b-7e94-42fe-a99e-dbd75fb00332/IHandle1b464420-82e0-45d3-970b-0c4c344cc770(omero.cmd._HandleTie@30297944)
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Closing...
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Handle closed before finished! State=READY
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,167 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Cancelling...
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) Cancelled
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) notify cancelled: null/omero.cmd.Status@4240ce1e
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,271 INFO  [ o.s.b.r.ManagedImportRequestI.@7d32954e] (l.Server-4) notify cancelled: null/omero.cmd.Status@4240ce1e
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,290 INFO  [    ome.formats.OMEROMetadataStoreClient] (2-thread-4) Pinging session every 300s.
/opt/omero/server/OMERO.server/var/log/Blitz-0.log.2:2023-12-14 11:46:16,306 INFO  [        ome.services.util.ServiceHandler] (l.Server-3)  Meth:       interface ome.api.StatefulServiceInterface.close

will-moore commented 8 months ago

Dom saw the same error for idr0138 NGFF import at https://idr-redmine.openmicroscopy.org/issues/266#note-20 Solved by not importing in parallel.

will-moore commented 3 months ago

4 months later...

We now have s3-support in ZarrReader, using a .zarr.bfoptions file as a sibling beside the .zarr directory (symlink) that contains the alt_store in addition to other flags that tell ZarrReader to optimise performance, e.g. making assumptions that all images are same dimensions etc.

omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://path/to/data.zarr`

Test import on pilot-idrngff which has recent ZarrReader AND Bio-Formats jars updated from daily build:

bash-5.1$ ls -alh /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_b45.jar 
-rw-r--r--. 1 omero-server omero-server 26M Apr 22 12:06 /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_b45.jar

bash-5.1$ ls -alh /opt/omero/server/OMERO.server/lib/server/ | grep SNAPSHOT
-rw-r--r--. 1 omero-server omero-server 143K Apr 25 15:57 formats-api-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 880K Apr 25 15:57 formats-bsd-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.5M Apr 25 15:57 formats-gpl-8.0.0-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server  14K Apr 25 15:57 metakit-5.3.8-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 7.8K Apr 25 15:57 mipav-stubs-6.0.3-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server  90K Apr 25 15:57 ome-codecs-1.0.4-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 139K Apr 25 15:57 ome-common-6.0.23-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.0M Apr 25 15:57 ome-jai-0.1.5-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server  90K Apr 25 15:57 ome-mdbtools-5.3.4-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 928K Apr 25 15:57 ome-poi-5.3.10-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 639K Apr 25 15:57 ome-xml-6.3.7-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 615K Apr 25 15:57 specification-6.3.7-SNAPSHOT.jar
-rw-r--r--. 1 omero-server omero-server 1.3M Apr 25 15:57 turbojpeg-8.0.0-SNAPSHOT.jar

Want to install aws..

$ sudo dnf install awscli
...
Installed:
  awscli-1.23.10-3.el9.noarch   groff-1.22.4-10.el9.x86_64             paper-2.3-6.el9.x86_64                perl-File-Copy-2.34-480.el9.noarch  perl-IPC-Run3-0.048-23.el9.noarch     perl-locale-1.09-480.el9.noarch   
  psutils-2.07-1.el9.noarch     python3-botocore-1.25.10-1.el9.noarch  python3-colorama-0.4.4-12.el9.noarch  python3-docutils-0.16-6.el9.noarch  python3-jmespath-0.9.4-11.el9.noarch  python3-pyasn1-0.4.8-6.el9.noarch 
  python3-rsa-4.9-2.el9.noarch  python3-s3transfer-0.5.2-2.el9.noarch 

Complete!

$ which aws
/usr/bin/aws

Download metadata-only plates as omero-server...

sudo -u omero-server -s
cd
mkdir idr0125 && cd idr0125
vi idr0125_plates.csv.    # added first 5 plates from list above

for zarr in $(cat idr0125_plates.csv);
  do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done

Import...

omero login
export OMERODIR=/opt/omero/server/OMERO.server

for zarr in $(cat idr0125_plates.csv);
  do omero import --depth=100 --transfer=ln_s --skip=all idr0125/$zarr --file /tmp/idr0125_$zarr.log  --errs /tmp/idr0125_$zarr.err;
done

log... First plate...

...
2024-04-30 14:34:27,973 13708373   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320647

==> Summary
11157 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 3:48:16.912

will-moore commented 3 months ago

Let's try to create a new bfoptions file for each Fileset as at https://github.com/IDR/idr-metadata/issues/684#issuecomment-1886332782 with the appropriate data For first Plate, with Fileset paths like: demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/I/21/5/0/.zarray

We want to generate .bfoptions like this for first Plate

omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

First Plate Fileset ID is 6320647. First Image is 15154987 Need to check which file is referred to in Pixels Table:

idr=> select path, name from Pixels where image=15154987;
                                                           path                                                           |       name       
--------------------------------------------------------------------------------------------------------------------------+------------------
 demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME | METADATA.ome.xml
(1 row)

As omero-server on pilot-idrngff

For ZarrReader, the bfoptions file is a sibling of the .zarr directory:

export bfoptionspath="/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions"
export fsid=6320647

echo "omezarr.list_pixels=false" > $bfoptionspath;
echo "omezarr.quick_read=true" >> $bfoptionspath;
echo "omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr" >> $bfoptionspath;

psql -U omero -d idr -h $DBHOST -f addbfoptions.sql 
CREATE FUNCTION

psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions', '$SECRET')"
 add_origfile_to_fileset 
-------------------------
                61623686
(1 row)

rm -f /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo

Try to view image... 10:40...

will-moore commented 3 months ago

Saving memo was very fast (300 ms!?):

(venv3) bash-5.1$ grep -B 3 "10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -B 3 "saved memo"
2024-05-01 09:40:32,659 INFO  [                      omero.cmd.SessionI] (l.Server-5) Removed servant from adapter: be265e12-20c1-45fb-bcdf-8695e703ddf5omero.api.IScript
2024-05-01 09:40:37,416 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo6657480071468204245
2024-05-01 09:40:37,416 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1714556437115] time[300] tag[loci.formats.Memoizer.saveMemo]
2024-05-01 09:40:37,416 DEBUG [                   loci.formats.Memoizer] (l.Server-7) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6406530 bytes)

Images are viewable! - approx 6 seconds to render a plane (we only have a single resolution level - added comment at https://github.com/broadinstitute/lincs-cell-painting/issues/54#issuecomment-2088292778)

Save To All resulted in many thumbnails being updated, but at ~6 seconds each this eventually timed-out, so we only have thumbnails for part of the plate:

Screenshot 2024-05-01 at 12 14 48

will-moore commented 3 months ago

To run the above bfoptions generation for "all" plates...

Need to generate csv with:

fileset ID: 6320647
templateprefix: demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/
zarr name: SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Used https://github.com/IDR/idr0125-way-cellpainting/blob/0a59dab08898e0deb4e75d5b6a80f36e94ac9315/scripts/get_filesets_to_csv.py to generate

idr0125_filesets.csv

6320647,SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-9/2024-04/30/10-46-12.114/
6320648,SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-3/2024-04/30/14-34-41.406/
6320649,SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-11/2024-04/30/18-29-40.187/
6320650,SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-8/2024-04/30/22-34-25.404/
6320651,SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr,demo_2/Blitz-0-Ice.ThreadPool.Server-0/2024-05/01/02-28-45.532/

psql -U omero -d idr -h $DBHOST -f addbfoptions.sql 
CREATE FUNCTION

export s3url="https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr"

for r in $(cat idr0125_filesets.csv); do
  fsid=$(echo $r | cut -d',' -f1)
  zarrname=$(echo $r | cut -d',' -f2)
  prefix=$(echo $r | cut -d',' -f3 | sed 's/.$//')
  bfoptionspath="/data/OMERO/ManagedRepository/$prefix/$zarrname.bfoptions"
  echo $bfoptionspath
  echo "omezarr.list_pixels=false" > $bfoptionspath
  echo "omezarr.quick_read=true" >> $bfoptionspath
  echo "omezarr.alt_store=$s3url/$zarrname" >> $bfoptionspath
  psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, '$zarrname.bfoptions', '$SECRET')"
  rm /data/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/$zarrname/OME/.METADATA.ome.xml.bfmemo
done

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2024-04/30/14-34-41.406/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions
 add_origfile_to_fileset 
-------------------------
                61623688
(1 row)

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-11/2024-04/30/18-29-40.187/SQ00014814__2016-05-23T17_24_56-Measurement1.ome.zarr.bfoptions
 add_origfile_to_fileset 
-------------------------
                61623689
(1 row)

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-8/2024-04/30/22-34-25.404/SQ00014815__2016-05-23T15_46_08-Measurement1.ome.zarr.bfoptions
 add_origfile_to_fileset 
-------------------------
                61623690
(1 row)

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-0/2024-05/01/02-28-45.532/SQ00014816__2016-05-23T14_07_55-Measurement1.ome.zarr.bfoptions
 add_origfile_to_fileset 
-------------------------
                61623691
(1 row)

Plate images are viewable 👍

will-moore commented 3 months ago

Copy rendering settings from image viewed above (settings applied in webclient) to the 5 plates imported to date...

Run in a screen...

for i in {10551..10555}; do
    echo "$i"
    omero --debug=2 render copy Image:15154987 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ tail -f /tmp/idr0125_copyplate_10551.err   
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Image:15154988 got thumbnail in 8.03s
Image:15154989 got thumbnail in 54.77s
Image:15154990 got thumbnail in 62.49s
Image:15154991 got thumbnail in 34.76s
Image:15154992 got thumbnail in 49.56s
...

will-moore commented 3 months ago

Rendering of thumbnails took ~16 hours for the first plate - approx 16 secs per image on average.

will-moore commented 3 months ago

First plate took ~16 hours, 2nd plate took 24 hours, then last 3 plates took about 12 hours each. This is comparable to above (before s3 support in ZarrReader) e.g. 13 hours at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1785591507 and after.

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ ls -alh /tmp/idr0125_copyplate_*
-rw-r--r--. 1 omero-server omero-server 130K May  2 05:57 /tmp/idr0125_copyplate_10551.err
-rw-r--r--. 1 omero-server omero-server 2.9K May  2 05:57 /tmp/idr0125_copyplate_10551.log
-rw-r--r--. 1 omero-server omero-server 131K May  3 06:12 /tmp/idr0125_copyplate_10552.err
-rw-r--r--. 1 omero-server omero-server 2.9K May  3 06:12 /tmp/idr0125_copyplate_10552.log
-rw-r--r--. 1 omero-server omero-server 130K May  3 18:06 /tmp/idr0125_copyplate_10553.err
-rw-r--r--. 1 omero-server omero-server 2.9K May  3 18:06 /tmp/idr0125_copyplate_10553.log
-rw-r--r--. 1 omero-server omero-server 130K May  4 07:27 /tmp/idr0125_copyplate_10554.err
-rw-r--r--. 1 omero-server omero-server 2.9K May  4 07:27 /tmp/idr0125_copyplate_10554.log
-rw-r--r--. 1 omero-server omero-server 129K May  4 16:27 /tmp/idr0125_copyplate_10555.err
-rw-r--r--. 1 omero-server omero-server 2.9K May  4 16:27 /tmp/idr0125_copyplate_10555.log

Checking the .log files, I see that all 3456 thumbnails are generated for all plates except first plate (1 less since the original image is there) and for the 2nd plate:

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ cat /tmp/idr0125_copyplate_10552.log
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 98 images.
Rendering settings successfully copied                               to 93 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 100 images.
Rendering settings successfully copied                               to 56 images.

This is missing 2 + 7 images: Browsing the data I can see missing thumbnails for Wells:

1O: last 2 fields
1P: last 7 fields

These Images all have Channels named e.g. 0, 1, 2, 3, 4 in webclient, imgData JSON has "label": "0",. This would explain why the copy of rendering settings failed above. However, the original NGFF data has: "label": "Channel 0" E.g. https://ome.github.io/ome-ngff-validator/?source=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/O/1/7/

The images appear to have "black" 0-array chunks (rather than missing chunks) of 9.2 kb each:

Screenshot 2024-05-07 at 11 53 45

We can manually fix this small number of images by copy, paste & save of rendering settings in webclient.

will-moore commented 3 months ago

Sample plate at https://ome.github.io/ome-ngff-validator/?source=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/ has downsampled resolutions...

Need to download and re-import metadata only plate etc...

aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

omero import --depth=100 --transfer=ln_s --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125_withdownscale.log --errs /tmp/idr0125_withdownscale.err

...
2024-05-17 16:44:19,239 17201442   [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61634458
2024-05-17 16:44:21,393 17203596   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/withdownscale/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320652

==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 4:46:17.439

will-moore commented 3 months ago

Create bfoptions...

vi /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-05/17/11-58-05.161/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions

omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions', '$SECRET')"

rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository//demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-05/17/11-58-05.161/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo

omero --debug=2 render copy Image:15172267 Plate:10556 > /tmp/idr0125_copyplate_20240519.log 2>/tmp/idr0125_copyplate_20240519.err

Looks faster!

Image:15172272 got thumbnail in 2.05s
Image:15172273 got thumbnail in 1.66s
Image:15172274 got thumbnail in 1.61s
Image:15172275 got thumbnail in 1.53s
Image:15172276 got thumbnail in 1.54s
Image:15172277 got thumbnail in 1.63s
Image:15172278 got thumbnail in 1.42s
...

EDIT: Done after approx 1.5 hours.

will-moore commented 2 months ago

After OME meeting - now we have pyramids for all plates

On pilot-idrngff, setup goofys mount of cellpainting-gallery so we can symlink to it after import. We won't have any other local or mounted copy of the data, so this will be essential for memo file generation (ZarrReader can use the public s3 link in bfoptions for subsequent pixel access, but this is seems not sufficient for future memo file generation).

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo mkdir /cellpainting-gallery
(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo goofys -o allow_other cellpainting-gallery /cellpainting-gallery/

Import ALL metadata-only plates as above

Create idr0125_plates.csv from above https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-1754891150

bash-5.1$ pwd
/opt/omero/server/idr0125
bash-5.1$ wc idr0125_plates.csv 
 136  136 7344 idr0125_plates.csv

In a screen...

for zarr in $(cat idr0125_plates.csv);
  do aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/$zarr idr0125/$zarr;
done

~6 hours later:

for zarr in $(cat idr0125_plates.csv);
  do omero import --depth=100 --transfer=ln_s --skip=all idr0125/$zarr --file /tmp/idr0125_$zarr.log  --errs /tmp/idr0125_$zarr.err;
done

will-moore commented 2 months ago

First import log

...
2024-06-05 01:30:06,683 15009142   [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/P/9/.zgroup
2024-06-05 01:45:27,401 15929860   [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2024-06-05 01:45:29,347 15931806   [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61659846
2024-06-05 02:22:04,606 18127065   [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61659846
2024-06-05 02:41:29,988 19292447   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61659846
2024-06-05 02:41:30,049 19292508   [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61659846
2024-06-05 02:41:30,085 19292544   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61659846
2024-06-05 02:41:30,114 19292573   [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61659846
2024-06-05 02:41:36,564 19299023   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/idr0125/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6320653

==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 5:11:36.992

Takes approx 5 hours instead of under 4 hours, probably due to extra resolution level

will-moore commented 2 months ago

For first 5 Plates, put these into a Screen:3502 so we can update symlinks... Using branch https://github.com/IDR/idr-utils/pull/54

cd /uod/idr/metadata/idr-utils/
$ sudo -Es git fetch will
$ sudo -Es git checkout will/managed_repo_symlinks

As omero-server

screen -S idr0125_symlinks
source ~/venv3/bin/activate
cd /data/idr-metadata/idr-utils

omero login

# check mount - should list plates
ls /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr

$ python scripts/managed_repo_symlinks.py Screen:3502 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report

Took less than 10 mins for 5 plates. Images are now viewable with chunks.

Rendering times (without bfoptions s3) in Preview panel is about 0.5 secs or less for low-resolution image plane (3 channels) and 2-3 seconds for full-resolution tiles.

Try setting rendering settings and thumbnails without bfoptions, for first 5 plates imported above... 16:49...

for i in {10557..10561}; do
    echo "$i"
    omero --debug=2 render copy Image:15175723 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done

Logs show 2-3 secs per Thumbnail...

Image:15175724 got thumbnail in 3.75s
Image:15175725 got thumbnail in 3.11s
Image:15175726 got thumbnail in 3.65s
Image:15175727 got thumbnail in 3.11s
Image:15175728 got thumbnail in 3.25s
Image:15175729 got thumbnail in 3.21s
Image:15175730 got thumbnail in 2.60s
Image:15175731 got thumbnail in 2.87s
Image:15175732 got thumbnail in 2.86s
Image:15175733 got thumbnail in 0.31s
Image:15175734 got thumbnail in 2.73s
Image:15175735 got thumbnail in 3.48s
Image:15175736 got thumbnail in 3.20s
Image:15175737 got thumbnail in 3.39s
Image:15175738 got thumbnail in 2.99s
Image:15175739 got thumbnail in 2.88s

...and 3 hours per plate:

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ ls -alh /tmp | grep copyplate
-rw-r--r--.  1 omero-server omero-server 129K Jun  6 18:46 idr0125_copyplate_10557.err
-rw-r--r--.  1 omero-server omero-server 2.9K Jun  6 18:46 idr0125_copyplate_10557.log
-rw-r--r--.  1 omero-server omero-server 129K Jun  6 21:46 idr0125_copyplate_10558.err
-rw-r--r--.  1 omero-server omero-server 2.9K Jun  6 21:46 idr0125_copyplate_10558.log
-rw-r--r--.  1 omero-server omero-server 129K Jun  7 00:44 idr0125_copyplate_10559.err
-rw-r--r--.  1 omero-server omero-server 2.9K Jun  7 00:44 idr0125_copyplate_10559.log
-rw-r--r--.  1 omero-server omero-server 129K Jun  7 03:49 idr0125_copyplate_10560.err
-rw-r--r--.  1 omero-server omero-server 2.9K Jun  7 03:49 idr0125_copyplate_10560.log
-rw-r--r--.  1 omero-server omero-server 129K Jun  7 06:53 idr0125_copyplate_10561.err
-rw-r--r--.  1 omero-server omero-server 2.9K Jun  7 06:53 idr0125_copyplate_10561.log

will-moore commented 1 month ago

Import still running from 3 weeks ago https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2147163019

Now we have 90 newly-imported Plates (in addition to the 5 above). Put them in a Screen in webclient and update symlinks as in previous comment:

python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
...
Fileset: 6320745 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-44/2024-06/24/23-22-39.294/
Render Image 15493675
fs_contents ['SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-44/2024-06/24/23-22-39.294/SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015165__2016-04-21T00_16_52-Measurement1.ome.zarr

Fileset: 6320746 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-41/2024-06/25/04-21-54.420/
Render Image 15497131
fs_contents ['SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-41/2024-06/25/04-21-54.420/SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015166__2016-04-21T01_55_54-Measurement1.ome.zarr

Fileset: 6320747 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-40/2024-06/25/09-28-38.106/
Render Image 15500587
fs_contents ['SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-40/2024-06/25/09-28-38.106/SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015167__2016-04-21T03_34_00-Measurement1.ome.zarr

will-moore commented 1 month ago

Previously we don't need to add the bfoptions to the Fileset (as we did for idr0009 and tested above) since ZarrReader will check for the .bfoptions file regardless (as it does for all other NGFF data)!

Test on one Plate... First of the 5 imported above: http://localhost:1080/webclient/?show=plate-10557

demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Need to create .bfoptions at

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr.bfoptions

omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr_withdownscale8/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Delete memo...

rm -f /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo

Taking a while to regenerate (viewing in webclient)...

Eventually done - logs claim 414 milli-secs - although this completed at 17:06BST (16:06 GMT) and was started some time ago!

grep -B 3 "21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -B 3 "saved memo"
2024-06-26 16:06:45,194 INFO  [                      omero.cmd.SessionI] (.Server-43) Removed servant from adapter: eaaf2e19-0308-470a-a645-13705799f767omero.api.IQuery
2024-06-26 16:06:48,659 DEBUG [                   loci.formats.Memoizer] (.Server-37) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo6092143311287441869
2024-06-26 16:06:48,659 DEBUG [                   loci.formats.Memoizer] (.Server-37) start[1719418008245] time[414] tag[loci.formats.Memoizer.saveMemo]
2024-06-26 16:06:48,659 DEBUG [                   loci.formats.Memoizer] (.Server-37) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/04/21-30-02.916/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996260 bytes)

Previously, when bfoptions file was added to the Fileset, memo file regeneration was very fast (300 ms)! https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2088306464 So let's try that with another Plate - 2nd of the bunch of 5... http://localhost:1080/webclient/?show=plate-10558

demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr

Create bfoptions at

/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions

with

omezarr.list_pixels=false
omezarr.quick_read=true
omezarr.alt_store=https://cellpainting-gallery.s3.amazonaws.com/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr

$ psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset(6320654, 'SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr.bfoptions', '$SECRET')"
 add_origfile_to_fileset 
-------------------------
                64161893
(1 row)

Delete memo...

rm /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/05/02-49-36.819/SQ00014813__2016-05-23T19_03_28-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo

View in webclient.... again, memo file generation is not quick....

will-moore commented 1 month ago

Copy rendering settings from first Image to the rest of first Plate...

omero --debug=2 render copy Image:15175723 Plate:10557 > /tmp/idr0125_copyplate_10557.log 2>/tmp/idr0125_PR59_copyplate_10557.err

will-moore commented 1 week ago

Seems that not all 90 plates in Screen above have had symlinks created correctly at https://github.com/IDR/idr0125-way-cellpainting/issues/4#issuecomment-2189178399 The 3 Plates listed there are OK (can see chunks loading in webclient) but others not working.

Try re-running:

$ ssh -A pilot-idrngff -L 1080:localhost:80
$ sudo -u omero-server -s
$ screen -r idr0125_symlinks
$ source /opt/omero/server/venv3/bin/activate

$ cd /uod/idr/metadata/idr-utils/

$ python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report
fileset_dirs {}

Fileset: 6320658 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2024-06/05/22-56-44.206/
Render Image 15193003
fs_contents ['SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-28/2024-06/05/22-56-44.206/SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014817__2016-05-23T12_08_18-Measurement1.ome.zarr

Fileset: 6320659 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/
Render Image 15196459
fs_contents ['SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr

Fileset: 6320660 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/06/08-42-04.670/
Render Image 15199915
fs_contents ['SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-27/2024-06/06/08-42-04.670/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014819__2016-05-23T08_51_51-Measurement1.ome.zarr

Fileset: 6320661 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-37/2024-06/06/13-32-34.540/
Render Image 15203371
fs_contents ['SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-37/2024-06/06/13-32-34.540/SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr to /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014820__2016-05-25T21_16_25-Measurement1.ome.zarr

Fileset: 6320662 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-34/2024-06/06/18-46-57.670/
Render Image 15206827...

While browsing webclient while still running, got DatabaseBusyException

DatabaseBusyException

``` Traceback (most recent call last): File "/opt/omero/web/venv3/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omeroweb/decorators.py", line 538, in wrapped retval = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omeroweb/decorators.py", line 597, in wrapper context = f(request, *args, **kwargs) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omeroweb/webclient/views.py", line 1798, in load_metadata_preview manager = BaseContainer(conn, **{str(c_type): int(c_id)}) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omeroweb/webclient/controller/container.py", line 117, in __init__ self.well = self.conn.getObject("Well", well) File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero/gateway/__init__.py", line 3225, in getObject result = self.getQueryService().findByQuery( File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero/gateway/__init__.py", line 5056, in __getattr__ obj = self._obj or self._getObj() File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero/gateway/__init__.py", line 4987, in _getObj self._obj = self._create_func() File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero/gateway/__init__.py", line 4963, in cf obj = getattr(self._conn.c.sf, self._func_str)() File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero_API_ice.py", line 758, in getQueryService return _M_omero.api.ServiceFactory._op_getQueryService.invoke(self, ((), _ctx)) Ice.UnknownException: exception ::Ice::UnknownException { unknown = ome.conditions.DatabaseBusyException: cannot create transaction at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy92.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:447) at ome.services.util.Executor$Impl.execute(Executor.java:392) at ome.services.blitz.impl.ServiceFactoryI.isGuest(ServiceFactoryI.java:571) at ome.services.blitz.impl.ServiceFactoryI.disallowGuest(ServiceFactoryI.java:585) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:438) at ome.services.blitz.impl.ServiceFactoryI.getByName(ServiceFactoryI.java:430) at ome.services.blitz.impl.ServiceFactoryI.getQueryService(ServiceFactoryI.java:309) at omero.api._ServiceFactoryTie.getQueryService(_ServiceFactoryTie.java:185) at omero.api._ServiceFactoryDisp.___getQueryService(_ServiceFactoryDisp.java:626) at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1303) at IceInternal.Incoming.invoke(Incoming.java:221) at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) at Ice.ConnectionI.dispatch(ConnectionI.java:1145) at Ice.ConnectionI.message(ConnectionI.java:1056) at IceInternal.ThreadPool.run(ThreadPool.java:395) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) at java.base/java.lang.Thread.run(Thread.java:829) } ```

will-moore commented 1 week ago

Re-ran

python scripts/managed_repo_symlinks.py Screen:3503 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240806.log

Looks good this time - All 94 Plates completed... in ~3 hours Script started at 6.40 BST, completed 9:30 (based on time of last link 8:30 GMT)

bash-5.1$ grep "Render Image" /tmp/idr0125_symlinks_20240806.log | wc
     94     282    2068

bash-5.1$ ls -lh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-38/2024-06/26/06-17-38.755/
total 0
lrwxrwxrwx. 1 omero-server omero-server 142 Aug  6 08:30 SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr -> /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015171__2016-04-21T16_22_35-Measurement1.ome.zarr

Repeat with remaining 37 plates in temp Screen:3504 - add to same log...

python scripts/managed_repo_symlinks.py Screen:3504 cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240806.log

EDIT (7th Aug) - Mistake in previous command - missed the leading / Running again (much faster since Images are rendered already)

$ python scripts/managed_repo_symlinks.py Screen:3504 /cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr --repo /data/OMERO/ManagedRepository --report >> /tmp/idr0125_symlinks_20240807.log

(venv3) bash-5.1$ grep "Link from" /tmp/idr0125_symlinks_20240807.log | wc
     37     185   11248

will-moore commented 1 week ago

Trying to view images in webclient to trigger memo file regeneration:

Taking a very long time, with occasional DatabaseBusy exceptions -> restart server...

Checking logs for memo saving times for one plate give 4 results, all starting within a couple of seconds and taking 5 hours! I wonder if clicking in webclient can trigger multiple memo generations in parallel? Probably better to use script.

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ grep -A 3 "SQ00014818__2016-05-23T10_30_25" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 3 "saved memo"
2024-08-08 03:32:16,840 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:16,840 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1723069481615] time[18455225] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:16,840 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 1575
2024-08-08 03:32:16,847 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-3) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml
--
2024-08-08 03:32:16,909 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:16,909 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1723069491280] time[18445628] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:16,909 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 0
2024-08-08 03:32:16,910 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1723069491275] time[18445635] tag[omero.call.success.ome.services.RenderingBean$12.doWork]
--
2024-08-08 03:32:17,098 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:17,099 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1723069481453] time[18455645] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:17,099 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 648
2024-08-08 03:32:17,145 INFO  [ome.services.sessions.state.SessionCache] (2-thread-3) Destroying session 00807f1d-2eeb-474f-8c81-2d0cdad62308 due to : Timeout
--
2024-08-08 03:32:17,160 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/.METADATA.ome.xml.bfmemo (6996491 bytes)
2024-08-08 03:32:17,160 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1723069491451] time[18445708] tag[loci.formats.Memoizer.setId]
2024-08-08 03:32:17,160 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-23/2024-06/06/03-51-10.329/SQ00014818__2016-05-23T10_30_25-Measurement1.ome.zarr/OME/METADATA.ome.xml Series: 1
2024-08-08 03:32:17,189 INFO  [        ome.services.util.ServiceHandler] (2-thread-3)  Executor.doWork -- ome.services.sessions.SessionManagerImpl.reload[d03aac19-dbb9-477f-8983-298029bc6a0a]

will-moore commented 1 week ago

Memo file generation: As omero-server user

omero hql --limit -1 --ids-only --style csv 'select MIN(field.image.id) FROM WellSample AS field where field.well.plate.id in (select link.child.id from ScreenPlateLink as link where link.parent.id=3503) GROUP BY field.well.plate' > idr0125_plates.txt

Remove first row, then

cut -d ',' -f2 idr0125_plates.txt | sed -e 's/^/Image:/' > idr0125_plate_ids.txt

screen -dmS cache parallel --eta -a idr0125_plate_ids.txt --results /tmp/memo_idr0125_20240808/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

Memo generation took about an hour for each Plate (first 3 were already done):

(base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ grep ok /tmp/memo_idr0125_20240808/1/**/**
/tmp/memo_idr0125_20240808/1/Image:15193003/stdout:ok: Pixels:15193003 Image:15193003 2.009263277053833 
/tmp/memo_idr0125_20240808/1/Image:15196459/stdout:ok: Pixels:15196459 Image:15196459 2.880405902862549 
/tmp/memo_idr0125_20240808/1/Image:15199915/stdout:ok: Pixels:15199915 Image:15199915 2.134599447250366 
/tmp/memo_idr0125_20240808/1/Image:15203371/stdout:ok: Pixels:15203371 Image:15203371 3689.8124442100525 
/tmp/memo_idr0125_20240808/1/Image:15206827/stdout:ok: Pixels:15206827 Image:15206827 3754.8339743614197 
/tmp/memo_idr0125_20240808/1/Image:15210283/stdout:ok: Pixels:15210283 Image:15210283 3674.200692176819 
...

and the last one completed about 7am on the 9th (today). Took approx 13 hours. 👍

will-moore commented 1 week ago

Copy rendering settings to 131 plates... Instead of for loop (above)...

for i in {10562..10692}; do
    echo "$i"
    omero --debug=2 render copy Image:15175723 Plate:$i > /tmp/idr0125_copyplate_$i.log 2>/tmp/idr0125_copyplate_$i.err
done

Let's try to use parallel... First make list of plates (all sequential IDs):

for i in {10562..10692}; do
    echo "Plate:$i" >> idr0125_ids_plates.txt
done

Then... using demo login...

export $PASS=secret

screen -dmS cache parallel --eta -a idr0125_ids_plates.txt --results /tmp/copyplate_idr0125_20240809/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u demo -w $PASS copy Image:15175723'

imagesc-bot commented 6 days ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-ngff-data-in-the-idr/98630/3

will-moore commented 5 days ago

for i in {10562..10692}; do grep "100 images" /tmp/copyplate_idr0125_20240809/1/Plate:$i/stdout | wc done

Not 34 in the following Plates:

10568 (33) - SQ00015043 P9 Field 1 10605 (33) - SQ00015120 P1 Field 6 10633 (33) - SQ00015148 P24 Field 6 10657 (33) - SQ00015173 O19 Field 2 10659 (33) - SQ00015195 B23 Field 5 10661 (33) - SQ00015197 P1 Field 1 10662 (32) - SQ00015198 P1 Fields 2-8, P21 Field 2 10671 (33) - SQ00015207 P1 (missing Well), O1 Fields 1, 3-9, 10672 (33) - SQ00015208 B22 Field 5

will-moore commented 5 days ago

Annotations...

(base) [wmoore@pilot-idrngff-omeroreadwrite metadata]$ sudo -Es git clone git@github.com:IDR/idr0125-way-cellpainting.git

omero-server

$ omero metadata populate --report --batch 1000 --file screenA/idr0125-screenA-annotation.csv Screen:3503