Add image spacing (and maybe orientation) awareness

[HastingsGreer]

One priority of MONAI is to excel at solving problems specific to medical imaging. In my opinion, the biggest pain point in medical imaging using python is converting back and forth between pixel coordinates (i, j, k) and physical coordinates (x, y, z). To that end, I think MONAI should consider storing location, spacing and orientation metadata with all of its images.

I see three routes to spacing awareness that MONAI could take: Use an existing medical imaging library such as ITK or SimpleITK, tacking on orientation, origin, and spacing metadata to numpy arrays, or resampling images to uniform spacing on ingest.

Using an existing library would be less work, and would mean that MONAI would come with all the registration, filtering, and segmentation capabilities of ITK. In particular, augmentations such as cropping, rotating, and filtering would be spacing aware for free. MONAI would want to depend on ITK 5.1 as it adds

Tacking on metadata to np.ndarray objects could allow simpler interoperation with other libraries such as scikit-learn or opencv, especially if our format for this metadata became a python standard the way that numpy arrays have. If we take this option, we should attempt to upstream this metadata format to pynrrd, itk, and nibabel. There is some discussion of this already, see https://discourse.itk.org/t/images-in-physical-space-in-python/2124. Specifically, origin, spacing, and direction fields would be added.

Resampling images to uniform sampling as an optional part of image preprocessing pipelines would be the simplest solution, as it would only require modifying the nifti image reading code, and not the neural network code.

Here are some real world problems that would be much easier with a spacing and orientation aware neural network library. I’ve separated them into two categories by difficulty. I would love to advertise MONAI as a truly medical neural network library, by showing that pythonic solutions to these challenges were available in the MONAI examples folder:

Low Hanging Fruit:

• Display a sagittal slice of an MRI with the correct aspect ratio
  • Most python libraries make this very hard, so while this sounds trivial, it would be instantly impressive to expert medical python users
• Resample a volume to have uniform spacing
  • This is the first step in most medical deep learning pipelines
• Use a CT scan and an MRI scan (with different spacings) of the same organ as two channels of input to a neural network

Tasks that would set MONAI far ahead of the crowd:

• Find the original coordinates of a pixel in an image that has been augmented with cropping and rotation
• Maintain correct spacing, orientation, and coordinates through upsampling, downsampling, and cropping layers
  • For example the original U-Net architecture produces a segmentation with a different origin that the original image, because it crops at each convolution. Tracking the exact displacement is a source of many bugs: a library that kept track of the transform associated with each layer automatically would reduce errors
• Understand LPS vs RAS coordinate systems, etc, and have well-documented functions for converting between them

[Nic-Ma]

Hi @HastingsGreer ,

Thanks very much for raising this topic and for your suggestions! It's very useful for a medical specific DL project.

1. Now we have loaded several metadata from Nifti: https://github.com/Project-MONAI/MONAI/blob/master/monai/data/nifti_reader.py#L42 Like: _affine, original_affine, as_closestcanonical. And all components can use them(for example: restore the properties when writing out model results).

2. Some transforms can unify the input data according to "spacing", already in the development list.

We certainly need to consider more complicated features later as you listed. Thanks.

[wyli]

agreed, as a starting point we could have transforms to convert arrays into user-specified spacing and orientation according to the original metadata. these transforms could be invertible so that we resume to the original coordinate system when writing network outputs