Questions on switching other models with your sampling method

[TianwenZhou]

Dear Authors, Thank you so much for your great work, as far as I am concerned, you only tried two types of pre-trained diffusion models in your work. Could you please shed some lights on ways to use other pretrained diffusion models (SD-v2.1 etc.) with your sampling method? Thank you again!

[yuanzhi-zhu]

Hi @TianwenZhou ,

Thank you for your interest. Actually, most of the previous work including DiffPIR, DPS, and DDRM, rely on diffusion UNets in image space (as image denoisers).

To release the generative power of large diffusion models such as SD, I recommend you have a look at https://arxiv.org/abs/2310.01110 & https://arxiv.org/abs/2311.15658 from @HJ-harry

[TianwenZhou]

Dear @yuanzhi-zhu , Thank you so much for your prompt reply, it is really helpful for me! As I was browsing through the pretrained diffusion models, I only found unconditional model trained on 256x256 ImageNet, while models with other resolutions(e.g. 512x512) are conditional. Given the fact that training such a huge model from scratch is unrealistic, I wonder if you could kindly offer the method to load pretrained conditional 512x512 diffusion model to your sampling process.

[yuanzhi-zhu]

unfortunately, to the best of my knowledge, there is no 512*512 image space diffusion model open-sourced.

[TianwenZhou]

So are there any ways that I can use a class-conditional 512x512 image space diffusion model with your sampling strategy?

[yuanzhi-zhu]

sure, assume that you know the class of the corrupted image, you can use the guided score as $p(x)$.

[TianwenZhou]

Thank you, I'm also curious about ways to test directly on low resolution images, rather than degrading the HR image following the given degradation process

[yuanzhi-zhu]

The Python script we provided is meant to evaluate DiffPIR on a given dataset with HR images for computing metrics.

To handle LR images directly, you have to modify the dataloader https://github.com/yuanzhi-zhu/DiffPIR/blob/main/main_ddpir.py#L38, and delete the metrics calculating related codes.

[TianwenZhou]

Thank you so much for your kind help and guidance! I'll definitely test these out