search

tlnagy / OMETIFF.jl

I/O operations for OME-TIFF files in Julia
Other
24 stars 7 forks source link

add lazy loading of larger files/split files #3

Closed tlnagy closed 4 years ago

tlnagy commented 7 years ago

Like NRRD.jl this package should really support mmapping because then it could open files much greater in size than the available memory.

This is going to require something more complex than NRRD.jl since the information will likely be split across multiple files and places inside each file. Therefore I'll need to do this lazily when actually requested because otherwise it is infeasible.

Prior art:

tlnagy commented 7 years ago

One difficulty is to quickly determine which file contains the relevant information given a tuple of indices. I might be able to use interval trees/augmented trees to annotate each file with the ranges of each dimension that are stored inside it. Then per-file, I'll have a tuple of ZCT(P?) indices mapped to the offset of the IFD data within that file. Then I'll have to compute the location of the data using the XY indices.

tlnagy commented 5 years ago

This is going to be a lot easier with the changes in #24. I now construct the following mapping:

ZTCP indices -> file, ifd offset

So from the index in the 6D array, I immediately have the file location and the offset for the IFD within the file. Thus, the main complication remaining would be figuring out how to handle slices along the X and/or Y axes.