Some issues related to inpainting

[Lecxxx]

Hello! Thank you for your excellent work! As mentioned in your paper，

“When the side-interpolater is kept during inference, MDT naturally enables the image inpainting ability.”

But when I reintroduced the side insulator, I found that the results I got were completely incorrect. May I ask how to design the correct method for introducing side interpolators, or is this just your speculation?

[gasvn]

We only did the visual inpainting experiment as a initial demo. As the early diffusion steps, we use the side-interpolators to predict the tokens of missing part. Then, we use standard diffusion steps in the following steps to generate the inpainted images.

[Lecxxx]

@gasvn Thank you for your patient answer. I have another issue that the model will keep three files during the training process:

• Opt600000.pt
• Model600000.pt
• Ema 0.9999 600000.pt

May I ask what are the differences between these three files? Which one should I choose when adding _"resumecheckpoint" during training and when testing?

[gasvn]

Opt stores some optmization related things, model is the current model weights, and eam is the mom weights. You can pass the Model600000.pt path to the resume_checkpoint. This framework is borrowed from the ADM, and you can check it for more details.

[Lecxxx]

@gasvn Thank you for your help! The problem has been perfectly resolved.