Supporting read and write with large nifti files

[korbinian90]

The use case is very large nifti files (~100GB) that don't fit into RAM: Reading parts, writing parts and creating new images on disk.

The mmap option is really nice for reading them!

I managed to write to the raw field by just changing the read option from "r" to "r+" in niread. The thing I am missing now is to create a large empty nifti file on the disk that I can open with niread as mmap and fill afterwards.

And thanks for the work on this Package. I am new to Julia, but this package is already more powerful than the MATLAB counterpart I was using before.

[Tokazama]

I'm going to give you a fairly comprehensive answer from my perspective because I also need to point some of my colleagues to an answer relative to handling large amounts of NIfTI data in Julia.

Current

If you need to directly write to a NIfTI format and know what you're doing I'd look here to figure out how to do it on your own. If it's not important to save the data as a NIfTI (i.e. you just need to save the raw array of an image) it may be easier to save results using something like HDF5 or Arrow (Arrow has really nice cross language compatibility if you're working with people that refuse to use Julia).

Future

We're currently working towards a more standard format for medical imaging data in Julia. This may involve a more modular version of the currently implemented I/O routines (see this issue). This work is currently in a very immature state but will be tested in the tokazama-imagemeta branch.

I've been using the "standard" NIfTI library as a reference to ensure there's a robust translation between the current standard and what we're working towards. I'm assuming you're use to a NIfTI library that reads images by chunks/blocks as is implemented in most software that uses the aforementioned NIfTI library. There's no promise this will directly translate to implementation of the feature you've requested, but it's likely it would at least ease implementation of it.

In the end I'm fairly confident that the issue of working with large imaging data in Julia will become more user friendly (well, as user friendly as out of memory manipulation can reasonably be). However, whether the public API for this will be implemented directly under this package or some meta package is currently unclear.