Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the eventbased image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions.
https://github.com/AHupuJR/EFNet/assets/35628326/b888bb41-33f9-46de-b1b6-50d5291bf4ed
Contributed by Xiaolei Gu, Xiao Jin (Jiaxing Research Institute, Zhejiang University), Yuhan Bao (Zhejiang University).
From left to right: blurry image, deblurred image, visualized events.
Feel free to connect me for potential applications.
This implementation based on BasicSR which is a open source toolbox for image/video restoration tasks.
python 3.8.5
pytorch 1.7.1
cuda 11.0
git clone https://github.com/AHupuJR/EFNet
cd EFNet
pip install -r requirements.txt
python setup.py develop --no_cuda_ext
Use GoPro events to train the model. If you want to use your own event representation instead of SCER, download GoPro raw events and use EFNet/scripts/data_preparation/make_voxels_esim.py to produce your own event representation.
GoPro with SCER: [ETH_share_link] [BaiduYunPan/code: 3wm8]
REBlur with SCER: [ETH_share_link] [BaiduYunPan/code:f6ha]
We also provide scripts to convert raw event files to SCER using scripts in ./scripts/data_preparation/. You can also design your own event representation by modify the script. Raw event files download:
GoPro with raw events: [ETH_share_link] [BaiduYunPan/TODO]
REBlur with raw events: [ETH_share_link] [BaiduYunPan/TODO]
prepare data
download the GoPro events dataset (see Dataset) to
./datasets
it should be like:
./datasets/
./datasets/DATASET_NAME/
./datasets/DATASET_NAME/train/
./datasets/DATASET_NAME/test/
train
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/train/GoPro/EFNet.yml --launcher pytorch
eval
python basicsr/test.py -opt options/test/GoPro/EFNet.yml
prepare data
download the REBlur dataset (see Dataset) to
./datasets
it should be like:
./datasets/
./datasets/DATASET_NAME/
./datasets/DATASET_NAME/train/
./datasets/DATASET_NAME/test/
finetune
python ./basicsr/train.py -opt options/train/REBlur/Finetune_EFNet.yml
eval
python basicsr/test.py -opt options/test/REBlur/Finetune_EFNet.yml
TODO
All the qualitative results can be downloaded through Google Drive:
@inproceedings{sun2022event,
title={Event-Based Fusion for Motion Deblurring with Cross-modal Attention},
author={Sun, Lei and Sakaridis, Christos and Liang, Jingyun and Jiang, Qi and Yang, Kailun and Sun, Peng and Ye, Yaozu and Wang, Kaiwei and Gool, Luc Van},
booktitle={European Conference on Computer Vision},
pages={412--428},
year={2022},
organization={Springer}
}
Should you have any questions, please feel free to contact leo_sun@zju.edu.cn or leisun@ee.ethz.ch.
This project is under the Apache 2.0 license, and it is based on BasicSR which is under the Apache 2.0 license. Thanks to the inspirations and codes from HINet and event_utils