Zongsheng Yue, Jianyi Wang, Chen Change Loy
Conference Paper | Journal Paper | Project Page | Video
:star: If ResShift is helpful to your images or projects, please help star this repo. Thanks! :hugs:
Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps.
![]()
resshift
can be created and activated with:conda create -n resshift python=3.10
conda activate resshift
pip install -r requirements.txt
or
conda env create -f environment.yml
conda activate resshift
You can try our method through an online demo:
python app.py
python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v3
python inference_resshift.py -i [image folder/image path] -o [result folder] --task bicsr --scale 4
python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_imagenet --scale 1
python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_face --scale 1
python inference_resshift.py -i [image folder/image path] -o [result folder] --task faceir --scale 1
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/realsr_swinunet_realesrgan256.yaml --save_dir [Logging Folder]
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/realsr_swinunet_realesrgan256_journal.yaml --save_dir [Logging Folder]
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/inpaint_lama256_imagenet.yaml --save_dir [Logging Folder]
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/inpaint_lama256_face.yaml --save_dir [Logging Folder]
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/faceir_gfpgan512_lpips.yaml --save_dir [Logging Folder]
Synthetic data for image super-resolution: Link
Synthetic data for natural image inpainting: Link
Synthetic data for face image inpainting: Link
Synthetic data for blind face restoration: Link
Reproduce the results in Table 3 of our NeurIPS paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v1 --chop_size 64 --chop_stride 64 --bs 64
Reproduce the results in Table 4 of our NeurIPS paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v1 --chop_size 512 --chop_stride 448
Reproduce the results in Table 2 of our Journal paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v3 --chop_size 64 --chop_stride 64 --bs 64
Reproduce the results in Table 3 of our Journal paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v3 --chop_size 512 --chop_stride 448
Reproduce the results in Table 4 of our Journal paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_imagenet --scale 1 --chop_size 256 --chop_stride 256 --bs 32
Reproduce the results in Table 5 of our Journal paper:
python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_face --scale 1 --chop_size 256 --chop_stride 256 --bs 32
Reproduce the results in Table 6 of our Journal paper (arXiv):
python inference_resshift.py -i [image folder/image path] -o [result folder] --task faceir --scale 1 --chop_size 256 --chop_stride 256 --bs 16
This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.
This project is based on Improved Diffusion Model, LDM, and BasicSR. We also adopt Real-ESRGAN to synthesize the training data for real-world super-resolution. Thanks for their awesome works.
If you have any questions, please feel free to contact me via zsyzam@gmail.com
.