ESTRNN & BSD

[ECCV2020 Spotlight] Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

[IJCV2022] Real-world Video Deblurring: A Benchmark Dataset and An Efficient Spatio-Temporal Recurrent Neural Network (Springer Link)

by Zhihang Zhong, Ye Gao, Yinqiang Zheng, Bo Zheng, Imari Sato

This work presents an efficient RNN-based model and the first real-world dataset for image/video deblurring :-)

Visual Results

Results on REDS (Synthetic)

Results on GOPRO (Synthetic)

Results on BSD (Real-world)

Beam-Splitter Deblurring Dataset (BSD)

We have collected a new real-world video deblurring dataset (BSD) with more scenes and better setups (center-aligned), using the proposed beam-splitter acquisition system:

The configurations of the new BSD dataset are as below:

Quantitative results on different setups of BSD:

Quick Start

Prerequisites

• Python 3.6
• PyTorch 1.6 with GPU
• opencv-python
• scikit-image
• lmdb
• thop
• tqdm
• tensorboard

Downloading Datasets

Please download and unzip the dataset file for each benchmark.

• BSD (Full BSD with RAW)
• GOPRO
• REDS

If you failed to download BSD from Google drive, please try the following BaiduCloudDisk version:
BSD 1ms8ms, password: bsd1
BSD 2ms16ms, password: bsd2
BSD 3ms24ms, password: bsd3

Training

Specify \<path> (e.g. "./dataset/") as where you put the dataset file.

Modify the corresponding dataset configurations in the command, or change the default values in "./para/paramter.py".

Training command is as below:

python main.py --data_root <path> --dataset BSD --ds_config 2ms16ms

You can also tune the hyper-parameters such as batch size, learning rate, epoch number (P.S.: the actual batch size for ddp mode is num_gpus*batch_size):

python main.py --lr 1e-4 --batch_size 4 --num_gpus 2 --trainer_mode ddp

If you want to train on your own dataset, please refer to "/data/how_to_make_dataset_file.ipynb".

Inference

Please download checkpoints of pretrained models for different setups and unzip them under the main directory.

Dataset (Test Set) Inference

Command to run a pre-trained model on BSD (3ms-24ms):

python main.py --test_only --test_checkpoint ./checkpoints/ESTRNN_C80B15_BSD_3ms24ms.tar --dataset BSD --ds_config 3ms24ms --video

Blurry Video Inference

Specify "--src \<path>" as where you put the blurry video file (e.g., "--src ./blur.mp4") or video directory (e.g., "--src ./blur/", the image files under the directory should be indexed as "./blur/00000000.png", "./blur/00000001.png", ...).

Specify "--dst \<path>" as where you store the results (e.g., "--dst ./results/").

Command to run a pre-trained model for a blurry video is as below:

python inference.py --src <path> --dst <path> --ckpt ./checkpoints/ESTRNN_C80B15_BSD_2ms16ms.tar

Citing

If you use any part of our code, or ESTRNN and BSD are useful for your research, please consider citing:

@inproceedings{zhong2020efficient,
  title={Efficient spatio-temporal recurrent neural network for video deblurring},
  author={Zhong, Zhihang and Gao, Ye and Zheng, Yinqiang and Zheng, Bo},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16},
  pages={191--207},
  year={2020},
  organization={Springer}
}

@article{zhong2023real,
  title={Real-world video deblurring: A benchmark dataset and an efficient recurrent neural network},
  author={Zhong, Zhihang and Gao, Ye and Zheng, Yinqiang and Zheng, Bo and Sato, Imari},
  journal={International Journal of Computer Vision},
  volume={131},
  number={1},
  pages={284--301},
  year={2023},
  publisher={Springer}
}