Inpainting using Denoising Diffusion Probabilistic Models
CVPR 2022 [Paper]
git clone https://github.com/andreas128/RePaint.git
pip install numpy torch blobfile tqdm pyYaml pillow # e.g. torch 1.7.1+cu110.
pip install --upgrade gdown && bash ./download.sh
That downloads the models for ImageNet, CelebA-HQ, and Places2, as well as the face example and example masks.
python test.py --conf_path confs/face_example.yml
Find the output in ./log/face_example/inpainted
Note: After refactoring the code, we did not reevaluate all experiments.
What are the blue parts?
Those parts are missing and therefore have to be filled by RePaint.
RePaint generates the missing parts inspired by the known parts.
How does it work?
RePaint starts from pure noise. Then the image is denoised step-by-step.
It uses the known part to fill the unknown part in each step.
Why does the noise level fluctuate during generation?
Our noise schedule improves the harmony between the generated and
the known part [4.2 Resampling].
Which datasets and masks have a ready-to-use config file?
We provide config files for ImageNet (inet256), CelebA-HQ (c256) and Places2 (p256) for the masks "thin", "thick", "every second line", "super-resolution", "expand" and "half" in ./confs
. You can use them as shown in the example above.
How to prepare the test data?
We use LaMa for validation and testing. Follow their instructions and add the images as specified in the config files. When you download the data using download.sh
, you can see examples of masks we used.
How to apply it to other images?
Copy the config file for the dataset that matches your data best (for faces aligned like CelebA-HQ _c256
, for diverse images _inet256
). Then set the gt_path
and mask_path
to where your input is. The masks have the value 255 for known regions and 0 for unknown areas (the ones that get generated).
How to apply it for other datasets?
If you work with other data than faces, places or general images, train a model using the guided-diffusion repository. Note that RePaint is an inference scheme. We do not train or finetune the diffusion model but condition pre-trained models.
How to design a new schedule?
Fill in your own parameters in this line to visualize the schedule using python guided_diffusion/scheduler.py
. Then copy a config file, set your parameters in these lines and run the inference using python test.py --conf_path confs/my_schedule.yml
.
How to speed up the inference?
The following settings are in the schedule_jump_params key in the config files. You can visualize them as described above.
t_T
, the total number of steps (without resampling). The lower it is, the more noise gets removed per step.jump_n_sample
to resample fewer times.start_resampling
.Do you have further questions?
Please open an issue, and we will try to help you.
Did you find a mistake?
Please create a pull request. For examply by clicking the pencil button on the top right on the github page.
The blue region is unknown and filled by RePaint:
Note: RePaint creates many meaningful fillings.
1) Face: Expressions and features like an earring or a mole.
2) Computer: The computer screen shows different images, text, and even a logo.
3) Greens: RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants.
4) Garden: From simple filling like a curtain to complex filling like a human.
Intuition of one conditioned denoising step:
1) Sample the known part: Add gaussian noise to the known regions of the image.
We obtain a noisy image that follows the denoising process exactly.
2) Denoise one step: Denoise the previous image for one step. This generates
content for the unknown region conditioned on the known region.
3) Join: Merge the images from both steps.
Details are in Algorithm 1 on Page 5. [Paper]
This work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Nvidia GPU grant.
This repository is based on guided-diffuion from OpenAI.