[FEA] A graphical debugger for image loader

[vikashg]

Is your feature request related to a problem? Please describe. This feature request is related to the debugging the input data loader in a meaningful and tractable manner.

Describe the solution you'd like As monai uses a suite of data loaders and applies the transformations before feeding the data to the neural network. It is also important to check if the data preprocessing is correct. As of now what we do is that we will write a few lines of code between our transformations and save the image as a nifty or a .png image to make. However, this is not an efficient practice.

A good solution would be, if there is a debug flag which can be flipped to plot the input data along with all the transformations in a single image. Something like this.

In such a manner the whole input pipeline can be visually debugged in one step.

Describe alternatives you've considered There are some alternatives that exist for example Tensorboard. But Tensorboard doesn't provide a timeline of images processing. This might not be such a big problem when it comes to regular computer vision images, but it is really important in medical imaging.

Additional context Maintaining a visual history of image processing (transformations) applied before inference should be useful for radiologist while interpreting the results. It will also have a visual appeal for debugging purposes for developers. I can work on it.

[MMelQin]

Thanks @vikashg for the issue.

For now, we could interleave the SaveImageD transform in the pre-tranforms and post-transforms to save the Numpy (actually the keyed torch tensors in the MONAI/torch dataset) into nii on disk (requiring moving tensors off the GPU device if used), each with an app configured postfix to distinguish the file names of the intermediate images.

This is no replacement for supporting stepping through the transforms and inspecting the image in real time, but can be used for debugging for now. I'd maybe create an example for it (in my app for another project, I did chain up multi SaveImageD in the post transforms just to get the intermediate images, so I know this approach works).

Also, for the rendering of the final result (mask) image along with input, there is the Render Server piece coming soon, and its volumetric rendering, even in cine mode, really blew me away! Beside, it supports DICOM too.

[MMelQin]

@vikashg I have quickly updated the UNETR example app, just to quickly show that interleaving SaveImageD in the pre-transforms would at least preserve the intermediate images when the app runs, as seen in the output below, followed by the code snippet to make it happen.

mqin@mingq-dt:~/src/monai-app-sdk/examples/apps/ai_unetr_seg_app$ python3 __main__.py -m models/model.ts
Going to initiate execution of operator DICOMDataLoaderOperator
Executing operator DICOMDataLoaderOperator (Process ID: 13654, Operator ID: 523f73ad-4a03-4a29-b3b8-b8c767cb3e85)
Done performing execution of operator DICOMDataLoaderOperator

Going to initiate execution of operator DICOMSeriesSelectorOperator
Executing operator DICOMSeriesSelectorOperator (Process ID: 13654, Operator ID: 82e7a748-643b-4128-ab8c-b82b6448c6e9)
Done performing execution of operator DICOMSeriesSelectorOperator

Going to initiate execution of operator DICOMSeriesToVolumeOperator
Executing operator DICOMSeriesToVolumeOperator (Process ID: 13654, Operator ID: 4cbb3d02-1365-48e9-a285-ad5987ddc09d)
Done performing execution of operator DICOMSeriesToVolumeOperator

Going to initiate execution of operator UnetrSegOperator
Executing operator UnetrSegOperator (Process ID: 13654, Operator ID: b3659a2d-379f-405a-8c2f-098e211e5d0f)
Operator output folder path: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/.monai_workdir/operators/b3659a2d-379f-405a-8c2f-098e211e5d0f/0/output/saved_images_folder
Operator publish folder (for intermediate images): /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish/Img_in_context/Img_in_context_postSpacingD.nii.gz.
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish/Img_in_context/Img_in_context_postOrientationD.nii.gz.
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish/Img_in_context/Img_in_context_postScaleIntensityRangeD.nii.gz.
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish/Img_in_context/Img_in_context_postCropForegroundD.nii.gz.
floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/.monai_workdir/operators/b3659a2d-379f-405a-8c2f-098e211e5d0f/0/output/saved_images_folder/Img_in_context/Img_in_context_seg.nii.gz.
file written: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/.monai_workdir/operators/b3659a2d-379f-405a-8c2f-098e211e5d0f/0/output/saved_images_folder/Img_in_context/Img_in_context.nii.gz.
Output Seg image numpy array shaped: (313, 429, 429)
Output Seg image pixel max value: 13
Done performing execution of operator UnetrSegOperator

Going to initiate execution of operator PublisherOperator
Executing operator PublisherOperator (Process ID: 13654, Operator ID: b81872e5-2a91-49cb-8a56-ad97befc5fda)
Mask file path: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/.monai_workdir/operators/b3659a2d-379f-405a-8c2f-098e211e5d0f/0/output/saved_images_folder/Img_in_context/Img_in_context_seg.nii.gz
Density file path: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/.monai_workdir/operators/b3659a2d-379f-405a-8c2f-098e211e5d0f/0/output/saved_images_folder/Img_in_context/Img_in_context.nii.gz
App publish folder: /home/mqin/src/monai-app-sdk/examples/apps/ai_unetr_seg_app/output/publish
Done performing execution of operator PublisherOperator

mqin@mingq-dt:~/src/monai-app-sdk/examples/apps/ai_unetr_seg_app$ ls output/publish/Img_in_context
Img_in_context_postCropForegroundD.nii.gz  Img_in_context_postOrientationD.nii.gz  Img_in_context_postScaleIntensityRangeD.nii.gz  Img_in_context_postSpacingD.nii.gz
mqin@mingq-dt:~/src/monai-app-sdk/examples/apps/ai_unetr_seg_app$ 

The (temp) code that makes saving intermediate images happen

    def pre_process(self, img_reader, out_dir: str="./prediction_output") -> Compose:
        """Composes transforms for preprocessing input before predicting on a model."""

        my_key = self._input_dataset_key
        return Compose(
            [
                LoadImaged(keys=my_key, reader=img_reader),
                AddChanneld(keys=my_key),
                Spacingd(keys=my_key, pixdim=(1.5, 1.5, 2.0), mode=("bilinear")),
                SaveImaged(keys=my_key, output_dir=out_dir, output_postfix="postSpacingD", output_dtype=uint8, resample=False),
                Orientationd(keys=my_key, axcodes="RAS"),
                SaveImaged(keys=my_key, output_dir=out_dir, output_postfix="postOrientationD", output_dtype=uint8, resample=False),
                ScaleIntensityRanged(my_key, a_min=-175, a_max=250, b_min=0.0, b_max=1.0, clip=True),
                SaveImaged(keys=my_key, output_dir=out_dir, output_postfix="postScaleIntensityRangeD", output_dtype=uint8, resample=False),
                CropForegroundd(my_key, source_key=my_key),
                SaveImaged(keys=my_key, output_dir=out_dir, output_postfix="postCropForegroundD", output_dtype=uint8, resample=False),
                ToTensord(my_key),
            ]
        )

[gigony]

@MMelQin

We can inherit MONAI's Compose class and override __call__ method to instrument a transform between existing transforms. (Creating a separate compose class that inherits Compose class for instrumentation)

https://docs.monai.io/en/latest/_modules/monai/transforms/compose.html#Compose.__call__

    def __call__(self, input_):
        for _transform in self.transforms:
            input_ = apply_transform(_transform, input_, self.map_items, self.unpack_items)
        return input_

[MMelQin]

I know, it is doable in a OO way, but given that there are many transforms, and the image capture/validation needs to be on specific transforms, having a blanket instrumentation may not work; for timing, maybe, but needs to be a noop at production time. As I said, I'd rather create a new transform or extension or other means to allow developer the flexibility to target specific transforms, defeatable for production of course.

There are also needs to support plugin for and launching the visualization module, so it is not just a simple instrumentation, otherwise a decorator (remember it is used in some forms of the transforms) would suffice.

[gigony]

@MMelQin Sounds great! It would be awesome to see the visualization :)

[vikashg]

@MMelQin I see the SaveImageD is a part of the composition if I understand correctly. This SaveImageD saves the transformed image at each step. This is what I would do also. The problem is that once tested the user needs to go back and delete all those lines, which can make the debugging operation a bit tedious.

[MMelQin]

Agree @vikashg . Using SaveImageD is just a stop-gap till we have the proper impl that cleanly integrates with visualization, perf analysis, image QA etc.

[gigony]

Related Issue/PR in MONAI Core:

• https://github.com/Project-MONAI/MONAI/issues/2721
• https://github.com/Project-MONAI/MONAI/pull/3039

There is another item named Interface to visualize the transform effect. MONAI Core team may invite us to the conversation.

[wyli]

(@rijobro @Nic-Ma for viz)

[gigony]

This one can be done with

• https://github.com/Project-MONAI/monai-deploy-app-sdk/issues/9