How can we use the LDM model of 'brain_image_synthesis_latent_diffusion_model' in unconditional manner?

[Ahn-Ssu]

Thanks for this amazing work, it helps a lot in trials and experiments.

We ran the given scripts of 'brain_image_synthesis_latent_diffusion_model' successfully. However, it seems that the given model with the pre-trained weights and configurations works for only conditions (i.e. the model always takes covariables, such as age, sex, and brain structure volumes).

Is there any way to use this model under uncondition (i.e. make a sample without any conditions)? We had tried to change some configurations related to 'conditioning', but None of them worked.

Thank you so much for your attention and participation.

[marksgraham]

Hi there.

I'm afraid in this specific case it won't be possible to sample without conditioning. This is because the model was trained in a slightly unconventional way. As well as providing the conditioning to the diffusion model in the forward pass using the context argument, conditioning is concatenated to the latents during every denoising step, as you can see here:

https://github.com/Project-MONAI/GenerativeModels/blob/894f2ec3452d1bedbcd72e9842e1fa1fb4535ce9/model-zoo/models/brain_image_synthesis_latent_diffusion_model/scripts/sampler.py#L29-L39

This means the conditioning is 'baked into' the network and there is no straightforward way to sample without it.

[Ahn-Ssu]

Hello, thank you for your response.

We have already reviewed the code script and attempted various ways to change conditions, but it didn't work. We have confirmed that the with_conditioning arguemtn for the diffusion model was passed as True, and the corresponding weights for that have been trained. Therefore, we have also verified that unconditioned sampling is not possible with the provided weights.

Our request is whether you can provide the weights for a model trained without conditioning? Specifically, for the unconditioned scenario used in the comparison with GANs-based methods in Subsection 3.1 Sampling Quality of the paper.

Thank you.

[marksgraham]

I'm afraid we don't have those weights available :(

The only suggestion I have is maybe you could initialise the conditioned model with the weights, do some 'surgery' to remove the parts of the model that relate to the conditioning, and fine-tune on a small dataset without conditioning.