Open sbesson opened 5 years ago
Looking into this a bit, I can't reproduce plane opening issues with the sample plate. A simple iteration over every plane:
import loci.formats.ImageReader;
public class Test {
public static void main(String[] args) throws Exception {
try (ImageReader reader = new ImageReader()) {
reader.setId(args[0]);
for (int s=0; s<reader.getSeriesCount(); s++) {
reader.setSeries(s);
for (int p=0; p<reader.getImageCount(); p++) {
/* debug */ System.out.println("opening plane " + p + " from series " + s);
reader.openBytes(p);
}
}
}
}
}
opens 4 planes per series across all 9240 series without an exception. Is the sample plate the same as the one that was imported and is throwing exceptions?
Assuming not, my initial guess is that there is at least one truncated file, which we've encountered in other formats but don't thoroughly handle in ColumbusReader
. https://github.com/openmicroscopy/bioformats/pull/3380 adds an extra check and relevant logging (which probably isn't a bad idea in any case), so that's at least something to try.
Sorry, realized that the last comment reflects testing with the wrong branch (https://github.com/openmicroscopy/bioformats/commits/develop). With https://github.com/idr/bioformats/commits/master, I can reproduce the original exception. I think https://github.com/IDR/bioformats/pull/11 will fix it, but https://github.com/openmicroscopy/bioformats/pull/3380 is probably still not a bad idea to have eventually.
Thanks @melissalinkert. I confirmed your statement by configuring one of the representative plates failing with IDR Bio-Formats and having it testing using the development Bio-Formats branch - see https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/5390/console. I assume the issue above should have failed the pixels test.
From https://github.com/openmicroscopy/bioformats/compare/v5.9.2...IDR:IDR-0.6.6, it looks like there was no upstream changes in ColumbusReader
as this reader was backported almost identically to the mainline Bio-Formats. My suspicion is that the issue above comes from the TIFF reading stack and was incidentally fixed during the Bio-Formats 6 work.
Thanks for opening both issues. Immediately, I will working on trying to merge the Bio-Formats 6 work into IDR Bio-Formats which hopefully should bring the fixes for this dataset. If this is not sufficient or the upgrade work ends up being much harder than initially planned, we can probably come back to https://github.com/IDR/bioformats/pull/11 for a fix specific to this study.
While working on https://github.com/IDR/idr0056-stojic-lncrnas/pull/1, @dominikl reported failures while applying the rendering settings:
Looking at the stack trace server-side, the error is thrown while trying to access individual planes:
The reader allowing to read data exported from a PerkinElmer Columbus system was introduced specifically in the context of two IDR studies:
id0019
andidr0020
. As can be seen from the description or representative samples, the reader currently expects one file containing the pixel data (TIFF) per well samples.In the case of
idr0056
, the dataset is structured such that all the well samples within one well are stored within a FLEX file. In order to be able to read the original submitted data, the reader needs to be updated to support this variant of the file format.Sample plate:
/uod/idr/filesets/idr0056-stojic-lncrnas/20190513-ftp/RNAi\ screenA/1921\[652\]/