search

IDR / bioformats

Bio-Formats is a Java library for reading and writing data in life sciences image file formats. It is developed by the Open Microscopy Environment (particularly UW-Madison LOCI and Glencoe Software). Bio-Formats is released under the GNU General Public License (GPL); commercial licenses are available from Glencoe Software.
http://www.openmicroscopy.org/site/products/bio-formats
GNU General Public License v2.0
0 stars 0 forks source link

Flex: Prevent divide by zero exception #18

Closed dgault closed 2 years ago

dgault commented 4 years ago

Fix for divide by zero ArithmeticException which has been reported on idr0072 (see https://idr-redmine.openmicroscopy.org/issues/116#note-29).

This minimal change should allow for all the plates to be imported without any exceptions

sbesson commented 4 years ago

Started the testing of this fix today within the context of IDR submissions. I used two plates generated by the Opera Flex system:

With the current production version of IDR Bio-Formats, running bin/omero import -f on both plates fails with an ArithmeticException.

After replacing the JARs under lib/client and re-running bin/omero import -f, both filesets are correctly detected by the FlexReader as a single HCS fileset with multiple plate acquisitions.

sbesson commented 4 years ago

The import of two failing plates of idr0072 using a server including this fix completed successfully.

sbesson commented 4 years ago

Trying to import one of the failing multi-acquisitions plates of idr0001 passed the initial candidates detection (Previously failing with ArithmeticException) but the import failed server side with the following error

bash-4.2$  /opt/omero/server/OMERO.server/bin/omero import --transfer=ln_s --skip=checksum --parallel-upload=8 -T Screen:name:idr0001-graml-sysgro/screenA_fix /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_01\(2012-12-14_22-00-15\)/008012001.flex 
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
2020-04-07 10:28:36,516 225        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.5.5
2020-04-07 10:28:36,530 239        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 0.6.3-SNAPSHOT revision: 584131bc3e47472fcea4c70b25d4a743055c27a3 date: 3 April 2020
...
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at ome.xml.model.Plate.getPlateAcquisition(Plate.java:587)
    at ome.xml.meta.OMEXMLMetadataImpl.setPlateAcquisitionWellSampleRef(OMEXMLMetadataImpl.java:12681)
    at ome.xml.meta.FilterMetadata.setPlateAcquisitionWellSampleRef(FilterMetadata.java:2553)
    at loci.formats.in.FlexReader.populateMetadataStore(FlexReader.java:664)
    at loci.formats.in.FlexReader.initFlexFile(FlexReader.java:582)
    at loci.formats.in.FlexReader.initFile(FlexReader.java:390)
    at loci.formats.FormatReader.setId(FormatReader.java:1389)
    at loci.formats.ImageReader.setId(ImageReader.java:849)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.Memoizer.setId(Memoizer.java:789)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at ome.services.blitz.repo.CheckedPath.bfSetId(CheckedPath.java:412)
    at ome.services.blitz.repo.ManagedImportRequestI.open(ManagedImportRequestI.java:763)
    at ome.services.blitz.repo.ManagedImportRequestI.init(ManagedImportRequestI.java:245)
    at omero.cmd.HandleI.steps(HandleI.java:438)
    at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
    at omero.cmd.HandleI$2.doWork(HandleI.java:383)
    at omero.cmd.HandleI$2.doWork(HandleI.java:380)
    at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy72.doWork(Unknown Source)
    at ome.services.util.Executor$Impl.execute(Executor.java:447)
    at omero.cmd.HandleI.run(HandleI.java:379)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at ome.services.util.Executor$Impl$1.call(Executor.java:488)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
, message=Index: 1, Size: 1}

    at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807) ~[omero-blitz.jar:5.5.5]
    at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334) [omero-blitz.jar:5.5.5]
    at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118) [omero-blitz.jar:5.5.5]
    at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145) [omero-blitz.jar:5.5.5]
    at IceInternal.Incoming.invoke(Incoming.java:221) [ice.jar:na]
    at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) [ice.jar:na]
    at Ice.ConnectionI.dispatch(ConnectionI.java:1145) [ice.jar:na]
    at Ice.ConnectionI.message(ConnectionI.java:1056) [ice.jar:na]
    at IceInternal.ThreadPool.run(ThreadPool.java:395) [ice.jar:na]
    at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na]
    at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) [ice.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]
2020-04-07 10:28:46,291 10000      [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.Exception: Import failure
    at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:701) ~[omero-blitz.jar:5.5.5]
    at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:354) [omero-blitz.jar:5.5.5]
    at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:328) [omero-blitz.jar:5.5.5]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_222]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_222]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]
2020-04-07 10:28:46,291 10000      [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Exiting on error

==> Summary
156 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:06.202
sbesson commented 4 years ago

Spent some additional time troubleshooting the failed import above. I originally missed the 156 files uploaded which should have given me some hint as the expectation is to have 575 files as in the original plate uploaded to IDR.

Increasing the debugging verbosity, the problem seems to come during the parsing of individual flex files. All flex files under Meas_01(2012-12-14_22-00-15) are correctly included in the used files. However, for the following plate acquisitions, only Flex files of the first row up to 001012001.flex are added to the used files.

...
2020-04-21 11:53:22,699 461632     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_06(2012-12-15_01-22-22)/001011001.flex
2020-04-21 11:53:22,699 461632     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,701 461634     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,701 461634     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,702 461635     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,704 461637     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 0)
2020-04-21 11:53:22,704 461637     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_01(2012-12-14_22-00-15)/001012001.flex
2020-04-21 11:53:22,704 461637     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,706 461639     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,706 461639     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,707 461640     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,710 461643     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 1)
2020-04-21 11:53:22,710 461643     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_02(2012-12-14_22-40-40)/001012001.flex
2020-04-21 11:53:22,710 461643     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,712 461645     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,712 461645     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,713 461646     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,714 461647     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 2)
2020-04-21 11:53:22,714 461647     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_03(2012-12-14_23-21-05)/001012001.flex
2020-04-21 11:53:22,714 461647     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,716 461649     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,716 461649     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,716 461649     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,717 461650     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 3)
2020-04-21 11:53:22,717 461650     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_04(2012-12-15_00-01-23)/001012001.flex
2020-04-21 11:53:22,717 461650     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,719 461652     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,719 461652     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,721 461654     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,722 461655     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 4)
2020-04-21 11:53:22,722 461655     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_05(2012-12-15_00-41-50)/001012001.flex
2020-04-21 11:53:22,722 461655     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,724 461657     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,724 461657     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,724 461657     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well A12
2020-04-21 11:53:22,725 461658     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well A12, field 5)
2020-04-21 11:53:22,725 461658     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_06(2012-12-15_01-22-22)/001012001.flex
2020-04-21 11:53:22,725 461658     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,727 461660     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,727 461660     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,728 461661     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B1
2020-04-21 11:53:22,730 461663     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B1, field 0)
2020-04-21 11:53:22,730 461663     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_01(2012-12-14_22-00-15)/002001001.flex
2020-04-21 11:53:22,730 461663     [      main] INFO                 loci.formats.FormatHandler - Parsing XML in .flex file
2020-04-21 11:53:22,732 461665     [      main] INFO                 loci.formats.FormatHandler - Populating pixel scaling factors
2020-04-21 11:53:22,732 461665     [      main] WARN                 loci.formats.FormatHandler - mismatch between image count, names and factors (count=18432, names=32, factors=32)
2020-04-21 11:53:22,732 461665     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B1
2020-04-21 11:53:22,735 461668     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B1, field 1)
2020-04-21 11:53:22,736 461669     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B1
2020-04-21 11:53:22,737 461670     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B1, field 2)
2020-04-21 11:53:22,737 461670     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B1
2020-04-21 11:53:22,738 461671     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B1, field 3)
2020-04-21 11:53:22,739 461672     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B1
2020-04-21 11:53:22,740 461673     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B1, field 4)
2020-04-21 11:53:22,741 461674     [      main] INFO                 loci.formats.FormatHandler - Parsing IFDs for well B2
2020-04-21 11:53:22,743 461676     [      main] INFO                 loci.formats.FormatHandler - Parsing .flex file (well B2, field 0)
2020-04-21 11:53:22,743 461676     [      main] DEBUG                loci.formats.FormatHandler - Parsing XML from /uod/idr/filesets/idr0001-graml-sysgro/20150928-disks1234/JL_121214_J2_1/Meas_01(2012-12-14_22-00-15)/002002001.flex
...

The same detection can be reproduced with the plates of idr0072 containing multiple Meas folders.