emilmelnikov
commented
10 months ago I've made some more changes.
Now HDF5 chunk size and imglib2 cell size are always separate. When reading chunked HDF5 file, it's chunk size no longer used as a cell size because chunk size is too small. When writing, cell size is chosen to be at most 256 MiB, which should be a good compromise between interactivity and performance. Write cell shape is now always either XY or XYZ with singleton dimensions for other axes.
When writing, callbacks now trigger status bar updates, so users can see the progress. Callback signature is changed: now it accepts long which is the total number of bytes written (it is easier to deal with bytes because we can precalculate the total number of bytes we need to read/write in advance, and use this for progress bar updates).
I think now this is really ready to be merged, if everything else is OK.
imagesc-bot
Previously, HDF5 datasets were always read and written plane-wise (XY). Also, pixel values were always boxed when reading. This resulted in sub-optimal performance, especially when the source HDF5 did not have XY as it's fastest changing dimensions.
The new reader implementation always reads data either in native HDF5 chunks, or in the largest possible contiguous blocks, and pixel values are never boxed. Moreover, if the source image is not chunked and sufficiently small (element count is less than 2^31), a simpler
ArrayImg(just a simple wrapper around a primitive array) is used instead ofCellImg(ND grid ofArrayImg).The writer still deals with boxed pixels: we need to be able to write (almost) arbitrary
ImgPluswhich could contain anything. However, it tries to minimize overhead by creating flat primitive arrays from large blocks obtained from the sourceImgPlusand write them directly. Note that these blocks are large and different from HDF5 chunks which are usually much smaller.Another important change is using image views both when reading and writing instead of trying to actually change data dimensions. When reading, source dimension order is preserved, but before returning it is (virtually) permuted to match the requested axis order. When writing, data is (also virtually) permuted before obtaining blocks and performing actual writes.
Additionally, support for other primitive data types has been added.