Port Existing Generative Tutorials

[ericspod]

Is your feature request related to a problem? Please describe. We have the following to port:

• [ ] 2d_autoencoderkl --> #1823
• [x] 2d_controlnet --> #1815
• [x] 2d_ddpm --> #1814
• [ ] 2d_diffusion_autoencoder
• [x] 2d_ldm --> #1775
• [x] 2d_spade_gan --> #1794
• [x] 2d_spade_ldm --> #1794
• [ ] 2d_super_resolution --> @virginiafdez
• [x] 2d_vqgan --> #1817
• [x] 2d_vqvae --> #1817
• [x] 2d_vqvae_transformer --> #1817
• [ ] 3d_autoencoderkl --> #1823
• [ ] 3d_ddpm --> @ericspod
• [x] 3d_ldm --> #1775
• [x] 3d_vqvae --> #1817
• [ ] anomaly_detection --> #1818
• [ ] classifier_free_guidance
• [ ] #1783
• [x] #1780
• [x] #1770

Let's divide amongst assignees and go from there. For each tutorial in the generative repo we should:

1. Copy over the .ipynb file and not the .py file made with papermill, only those .py files that were written as such (I think distributed_training/ddpm_training_ddp.py is the only one).
2. Change references from components in GenerativeModels to those in MONAI.
3. Verify the tutorial still works.
4. Fix any bugs.

[virginiafdez]

Most of the tutorials merely require the changes mentioned above except for the following:

• 3d_brainmask_controlnet: this tutorial is using synthetic 3D images generated by the model zoo LDM. The reason why we did that is that ControlNet requires to train an autoencoder, a diffusion model and then the controlnet network. In 3D, this is cumbersome - especially from a notebook tutorial. To shortcut this, we used the pre-trained LDM model (which includes the autoencoder and diffusion models), so that we only need to train the controlnet bit. However, this model is trained on T1 images. To have something to condition on, we decided to use the easiest form of condition we could do for the synthesis of T1 images, which are just brain masks. These masks and the specific training set we had to get them from somewhere, hence the current set-up where we use images from a Drive folder. Though implying the tutorial would change substantially, we could re-design this experiment. Perhaps we could train a very low dimension AE, LDM and Controlnet on BraTS.
• 2d_spade_gan and 2D_spade_ldm: SPADE works better when a lot of semantic coverage is offered. This makes BraTS not a suitable dataset for it, cause it has only tumour segmentation masks. This prompted the use of another small version of a dataset for which we had associated segmentations (of GM, WM, DGM etc.) generated with GIF. Nonetheless, this dataset is not straightforward open-source, so we should not use it. I suggest using BraTS instead. I have a 2mm isotropic version of 25 T1 volumes from BraTS with associated semantic maps that can be used. It would be hosted from my Drive, at least from early on. It occupies approximately 25MB. I've already checked it works.

[Nic-Ma]

Hi @ericspod ,

As our ETA is end of this month, could you please help split the tasks directly based on the effort in your mind? @virginiafdez @yiheng-wang-nv should be the experts to help here.

Thanks in advance.

[virginiafdez]

I will take on the porting of 2d_spade_gan and 2D_spade_ldm, as well as 3d_brainsmask_controlnet from August 17th onwards, as per what I discussed in my previous comment.

[Nic-Ma]

I will take on the porting of 2d_spade_gan and 2D_spade_ldm, as well as 3d_brainsmask_controlnet from August 17th onwards, as per what I discussed in my previous comment.

Hi @yiheng-wang-nv , could you please help also pick several of the tutorials to port?

Thanks in advance.

[ericspod]

I'm going to start with the 2D and 3D LDM tutorials in GenerativeModels. These are single notebooks which differ significantly in content from the existing LDM tutorials (https://github.com/Project-MONAI/tutorials/tree/main/generative/2d_ldm and https://github.com/Project-MONAI/tutorials/tree/main/generative/3d_ldm), so we have 4 tutorials here to check. I'll make sure the ones from GenerativeModels work first then those already here, and see how to best combine both.

[yiheng-wang-nv]

I will work in the order of:

• realism_diversity_metrics
• image_to_image_translation
• distributed_training -...

Will update this ticket when having any progrsses

[Nic-Ma]

Hi @ericspod ,

Thanks for your testing here, may I know why you want to test the "2D and 3D LDM tutorials in GenerativeModels"? I think maybe we can start the MONAI tutorials directly?

Thanks.

[ericspod]

may I know why you want to test the "2D and 3D LDM tutorials in GenerativeModels"?

These should be fast to check and then move over, the ones in tutorials are move involved and I need to go over them at the same time to be sure they aren't essentially duplicating what is being demonstrated.

[pedrob37]

I'll start working on anomaly_detection for now, and I'll likely get around to a few more next week

[ericspod]

I'm going to pick up the other 2D tutorials, hopefully these will be fast. the 3D DDPM tutorial works but I can't get good results out of it for some reason so far.