MedIO for MONAI

[jond01]

Dear MONAI members,

Your initiative looks very promising. I noticed that you are currently working on the IO part of the project: https://github.com/Project-MONAI/MONAI/wiki/IO-Working-Group-Meeting-Notes

We have recently released a python package medio whose objective is to facilitate the IO procedure for deep learning of medical images: https://github.com/RSIP-Vision/medio Medio uses ITK, NiBabel and PyDicom (including dicom-numpy for dicom series) in a uniform and consistent interface (metadata, orientations, ...).

I believe that it can really help here, what do you think?

[Nic-Ma]

Hi @aylward and @mjorgecardoso ,

I labeled this ticket as working group, maybe you can raise to working group discussion. Thanks.

[wyli]

we have been discussing a factory pattern which is similar to MedIO (as I understand it). Would you be happy to contribute the relevant logic into monai? we could continuously provide proper maintenance of this IO module. cc @ericspod @atbenmurray

[mjorgecardoso]

@wyli, fully agree. The IO wrapper/factory pattern is something we want to have WITHIN monai and not as an external dependency, as it will be highly intertwined with the transformation and data representation stacks.

@jond01, MONAI is trying to bring together and integrate multiple open source initiatives into the same stack. We are currently in the process of abstracting our IO stack to have some of the functionality you have in MedIO.

It would be great to discuss the feasibility of merging the MedIO package (or parts of it) into MONAI. Given that MedIO has been developed by a single user, code re-licensing and integration would be possible. Would be great to have RSIP-Vision as one of the many formal contributors to MONAI. (see the Contributors section of https://monai.io)

[jond01]

Thanks for your quick responses.

My original idea was integrating MedIO into MONAI just like you have integrated NiBabel. Keeping all the IO code inside the project sounds a bit challenging. I guess that at some point you would like to use ITK and PyDicom in addition to NiBabel, in order to support more general medical images input into MONAI.

In MedIO we faced various issues that you would also face here when integrating multiple packages:

1. Different conventions: nibabel uses nifti convention whereas itk and pydicom use dicom convention for the coordinate system (the translation between them is x->-x, y->-y, z->z). Moreover, the orientation ("axcodes") 'RAI' in nibabel would translate to 'LPS' in itk, and vise versa.
2. Itk code in python is nontrivial: itk is originally written in C++, and its python version is not very pythonic. There are even some open issues/merged PRs which address this in itk's repository. One more point - efficient reorientations in itk are tricky (efficient=no copy unless needed).
3. Affine matrix computation for pydicom: also included in medio (both multi-frame dicom and dicom series). Unlike nibabel, it is not just trivially there
4. 3d RGB images: applicable to nifti, dicom, and of course png/jpeg etc. The different representations should be consistent, and they are not such unless treated with care (for example, what axis is the channels axis?).
5. Dicom series data - reading and writing

The above points are just few examples...

I think that it should not be intertwined with other transforms in MONAI. For example, when reading you provide flags: backend='itk'/'nib'/'pydicom', desired_ornt='RAI'/'PLS', channels_axis=0/-1 and that's it for the reader part. What you get is a numpy array + metadata object including affine matrix, and can continue independently from there (rotating, zooming, normalizing intensity, etc.).

[mjorgecardoso]

Hi Jonathan,

One of the many aims within MONAI is to have a small dependency graph. Currently, it's only torch and numpy, and adding MedIO to the dependency tree would add MedIO and 5 other sub-dependencies. We are in the process of integrating python-ITK into our dependency tree, with several other backends as optional - thus the reason we want to have the IO backend wrapper within MONAI and not as an external dependency.

There is a long requirement list for IO that the IO_working_group has collected that go significantly beyond several of the points you expose above, which will require deep integration into the rest of the MONAI codebase. This is why @wyli was suggesting the integration/donation of some of the logic and code into the project MONAI repository, all with appropriate recognition of contribution.

[wyli]

just wanted to add my understanding that MedIO looks very promising in unifying thirdparty IO interfaces, but at the moment it's also a bit risky for monai to rely on it because its roadmap/contributors' commitment/code quality assurance are unclear. On the other hand, would be very nice to see RSIP-Vision joining project monai and we'll have much lower R&D risks because monai has the open and active support from the community of researchers/engineers https://monai.io/about.html

[jond01]

I definitely understand your point @wyli and @mjorgecardoso. Just to say, medio is already effective in multiple closed-source projects - and yet your conservative approach is legitimate.

I think that the future IO part of monai can be useful for its own, and better be disentangled from monai's main repository. Let me suggest the following: the IO part of monai will be a distinct project, for example - https://github.com/Project-MONAI/monaio It can be integrated into monai's main repository (https://github.com/Project-MONAI/MONAI) as a submodule, therefore no extra installation will be required apart from nibabel, itk, pydicom and maybe some more.

"monaio" can start from a fork of medio if you like (maybe even with the same name?).

Regarding the dependencies in medio, they are minimal: 1) numpy 2) ITK 3) NiBabel 4) PyDicom

2-4 depends at most on numpy and packaging.

5) dicom-numpy - depends on numpy and pydicom

---

Edit: Linking some related issues I've just found:

135

327 (maybe itk instead of simple-itk?)

604

[aylward]

Hi,

Interesting idea! We talked about aspects of this in the I/O working group, and we encountered several challenges: 1) We could not agree on a dicom I/O package, but we could all agree that no single dicom package worked for everyone: gdcm, pydicom, and others were discussed and rejected as complete solutions 2) Extensibility was important: I/O is not just images (it includes bounding boxes, demographics, etc) and I/O formats in other domains (pathology, etc) are beyond what is currently support by any one package (beyond even ITK). 3) Reproducibility was critical: It is important to be able to specify what version of a reader/library should be used to replicate an experiment. Sometimes changing an IO library or the version of an IO library can change results.

It seems like several of those challenges were also motivation for your work!

From those, we came up with some guiding principles for what we want from an I/O solution: (a) don’t reinvent the wheel, (b) focus MONAI's energy and contributions on deep learning methods, not on supporting I/O methods, and (c) provide clinical relevance

We are proposing an object factory mechanism that uses ITK and that can be extended to include other libraries on an experiment-by-experiment basis - much like what MedIO has proposed - and we will look at it for further inspiration. Also, the good news is that MONAI is already implementing an object factory mechanism for other purposes, and we will be able to build on that code for an object factory for IO, without introducing any additional dependencies.

We hope to post the proposal in the next day or so. Would greatly appreciate your feedback on it, and perhaps some of the MedIO code base will help speed our development.

Thank you! Stephen

[jond01]

[...]. On the other hand, would be very nice to see RSIP-Vision joining project monai and we'll have much lower R&D risks because monai has the open and active support from the community of researchers/engineers https://monai.io/about.html

Hi, it can probably work. We can contribute to your IO part according to the proposal (#856), let me know @wyli @mjorgecardoso.

[wyli]

[...]. On the other hand, would be very nice to see RSIP-Vision joining project monai and we'll have much lower R&D risks because monai has the open and active support from the community of researchers/engineers https://monai.io/about.html

Hi, it can probably work. We can contribute to your IO part according to the proposal (#856), let me know @wyli @mjorgecardoso.

thank you, would be great to have your contribution. @Nic-Ma and I will create new tickets to properly track our intention and progress towards #856