📖 Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution

[Paper]   [Supp]   [Poster]

Long Sun, Jiangxin Dong, Jinhui Tang, and Jinshan Pan
IMAG Lab, Nanjing University of Science and Technology

An overview of the proposed SAFMN. SAFMN first transforms the input LR image into the feature space using a convolutional layer, performs feature extraction using a series of feature mixing modules (FMMs), and then reconstructs these extracted features by an upsampler module. The FMM block is mainly implemented by a spatially-adaptive feature modulation (SAFM) layer and a convolutional channel mixer (CCM).

News

• [2024-07-16] Add 🤗HuggingFace online demo!
• [2024-05-08] Add the light_SAFMN++, which is the 1st place winner of the fidelity track of the Real-time 4K Super Resolution Challenge for compressed AVIF images, and is invited to give an oral presentation at the AIS2024 workshop.
• [2024-03-27] Add the improved ESR model SAFMN++, which ranks Top4 in the Overall Performance track and Runtime track of NTIRE2024 ESR challenge
• [2024-03-16] Add SAFMN_BCIE, which enhances the quality of JPEG images compressed with a large range of quality factors. The corresponding inference code can be found here.
• [2023-11-22] The code for ONNX export is available here
• [2023-09-08] Add :panda_face: OpenXLab online demo!
• [2023-08-31] Update SAFMN_Real_x4.pth
• [2023-08-31] Add SAFMN_Real_x2.pth
• [2023-08-21] Colab demo for SAFMN on x4 real-world image SR is available
• [2023-07-14] Our SAFMN is accepted to ICCV 2023
• [2023-06-06] The report of NTIRE 2023 Challenge on Efficient Super-Resolution is available here
• [2023-04-17] The SAFMN ranks Top6 for overall performance in the NTIRE2023 ESR challenge.
• [2023-04-17] The SAFMN variant ranks Top3 for model complexity in the NTIRE2023 ESR Challenge.
• [2023-03-22] The code and checkpoint for the NTIRE2023 Efficient Super-Resolution Challenge is available here.
• [2023-03-22] The pre-trained model with high-order degradation on the LSDIR dataset is available.
• [2023-03-13] The source codes, checkpoints and visual results are available.
• [2023-02-26] The paper is available here.

Requirements

• Python 3.8, PyTorch >= 1.11
• BasicSR 1.4.2
• Platforms: Ubuntu 18.04, cuda-11

Installation

# Clone the repo
git clone https://github.com/sunny2109/SAFMN.git
# Install dependent packages
cd SAFMN
pip install -r requirements.txt
# Install BasicSR
python setup.py develop

You can also refer to this INSTALL.md for installation

Training

Run the following commands for training:

# train SAFMN for x4 effieicnt SR
python basicsr/train.py -opt options/train/SAFMN/train_DF2K_x4.yml
# train SAFMN for x4 classic SR
python basicsr/train.py -opt options/train/SAFMN/train_L_DF2K_x4.yml

Testing

• Download the pretrained models.
• Download the testing dataset.
• Run the following commands:

  # test SAFMN for x4 efficient SR
  python basicsr/test.py -opt options/test/SAFMN/test_benchmark_x4.yml
  # test SAFMN for x4 classic SR
  python basicsr/test.py -opt options/test/SAFMN/test_L_benchmark_x4.yml
  # test SAFMN for x4 real-world SR (without ground-truth)
  python basicsr/test.py -opt options/test/SAFMN/test_real_img_x4.yml
  # test SAFMN for x4 real-world SR (large input)
  python inference/inference_real_safmn.py --input test_demo --output results/test_demo --scale 4 --large_input 

• The test results will be in './results'.

Results

• Pretrained models and visual results

  We have provided three ways to download our checkpoints.

  • 1.Download via the Google Drive links shown below.
  • 2.Download via the Baidu Netdisk links shown below.
  • 3.Visit our huggingface repo at https://huggingface.co/Meloo/SAFMN/tree/main and click the download icons.

• Results on Blind Compressed Images

• Results on AIGC Image

• Real-world SR Results

Real-World Image (x4)	Real-ESRGAN	SwinIR	SAFMN (ours)

• Efficient SR Results

  

• Classic SR Results

  

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{sun2023safmn,
    title={Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution},
    author={Sun, Long and Dong, Jiangxin and Tang, Jinhui and Pan, Jinshan},
    booktitle={ICCV},
    year={2023}
 }

Acknowledgement

This code is based on BasicSR toolbox. Thanks for the awesome work.

Contact

If you have any questions, please feel free to reach me out at cs.longsun@gmail.com