Filepatternreader memoizer fixes

IDR / bioformats

Bio-Formats is a Java library for reading and writing data in life sciences image file formats. It is developed by the Open Microscopy Environment (particularly UW-Madison LOCI and Glencoe Software). Bio-Formats is released under the GNU General Public License (GPL); commercial licenses are available from Glencoe Software.

http://www.openmicroscopy.org/site/products/bio-formats

GNU General Public License v2.0

0 stars 0 forks source link

Filepatternreader memoizer fixes #9

Closed sbesson closed 5 years ago

sbesson commented 5 years ago

The testing of ebea1f67d833e3dd2826774690f4e2f5b4cc2d48 deployed on test60 revealed that most current published IDR studies could have their memo files regenerated and read correctly except for the studies that use a FilePatternReader strategy (idr0010, idr0026, idr0045).

This reverts a set of additions introduced in #1 and should allow to import pattern-based datasets including the memoization of constituent files.

sbesson commented 5 years ago

Definitely some progress as I have been able to take the latest IDR OMERO, update the formats-bsd JARs using this branch and I was able to:

successfully import https://idr.openmicroscopy.org/webclient/?show=plate-4503
access the planes via the viewers (note colors are inverted since this is pre-rendering settings)

Two blockers

the logs contain a lot of 2018-11-13 13:42:20,613 WARN [ loci.formats.Memoizer] (.Server-12) minimumElapsed: wrong type (expected: java.lang.Long) which points to the Memoizer.wrap() used both by ScreenReader and FilePatternReader
I do not see the memo files for individual files being saved on disk so there are still performance issues for these readers

sbesson commented 5 years ago

With the last change I was able to re-import the plate from idr0010. Import was performed with --skip=all and created the .pattern memo file in the cache. Then accessing the data e.g. via the clients led to calling openBytes which in turned cached the .stk files.

Looks like we still need to review how to create all memo files in the very setId call. At the moment, the Memoizer.wrap() in the FilePatternReader constructor is a no-op (since the reader has not been decorated with a Memoizer yet and the options are probably empty.

Barring objections, I propose to get this as 0.5.2, deploy it on test60 and retest pattern-based studies.