Add support for reading compressed NDPI tiles

melissalinkert commented 2 months ago

This expands on the "precompressed" features added in #3992.

The main goal was to just add compressed tile reading to the NDPI reader, so that conversion from NDPI to DICOM wouldn't require much if any recompression. This required a variety of changes not limited to NDPI though:

update JPEGTurboService so that the repackaged compressed tiles can be retrieved, as well as the tile size and counts
move general TIFF tile copying logic out of SVSReader, and into MinimalTiffReader (to prevent duplicate logic in NDPIReader)
- this should also make it easier to support precompressed tiles in other TIFF-based formats later on
update NDPIReader to report optimal sizes based on actual IFD/JPEG stream, instead of constant 1024x1024
update DicomWriter to allow different tile sizes in each resolution, since NDPI reports different sizes per resolution
update NDPIReader to use the saveCompressedBytes API

The end result is that something like this should now work:

bfconvert -noflat -precompressed -compression JPEG CMU-1.ndpi cmu.dcm

and should complete within a few seconds as all tiles can be copied without any recompression.

I would not expect SVS -> DICOM support to change, but it probably makes sense to double-check just in case the refactoring caused any problems.

I don't expect this to fail tests, but excluding for now so this doesn't distract from 7.3.0. Given the API changes in JPEGTurboService, this probably does require a major version.

cc @dclunie, @fedorov

melissalinkert commented 1 week ago

I believe the exceptions in https://github.com/ome/bioformats/pull/4181#pullrequestreview-2130154564 are all fixed by 23780de.

melissalinkert commented 4 days ago

Loading the generated DICOM files into OMERO for visual assessment and comparison to the original, all datasets worked as expected with the exception of the DICOM dataset created from the SVS file using -precompressed and -compression JPEG-2000 as the import failed with

In this case the pyramid is converted tile-wise, but the extra images are not, which was a problem since the reader's optimal tile sizes are now sent to the writer any time -precompressed is used. I think 4f2a96e326 is one way to fix, but open to other thoughts.

imagesc-bot commented 3 days ago

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/file-format-to-store-images-using-ngff-coverter/98320/4

ome / bioformats

Add support for reading compressed NDPI tiles #4181