This is the official repository of DeMFI (Deep Joint Deblurring and Multi-Frame Interpolation).
[ArXiv_ver.] [ECCV2022_ver.] [Supp.] [Demo] [Poster] [Video5mins(YouTube)]
Last Update: 10 JULY 2022 - This work has been accepted to ECCV2022, we have uploaded a camera ready version (ECCV format) and a supplementary material in above links. Please note that the overall architecture and main experiments are same as the initial arxiv version.
Jihyong Oh and Munchurl Kim "DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting", European Conference on Computer Vision, 2022.
BibTeX
@inproceedings{Oh2022DeMFI, title={DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting}, author={Oh, Jihyong and Kim, Munchurl}, booktitle={European Conference on Computer Vision (ECCV)}, year={2022} }
If you find this repository useful, please consider citing our paper.
The 30fps blurry input frames are interpolated to be 240fps sharp frames. All results are encoded at 30fps to be played as x8 slow motion and spatially down-scaled due to the limit of file sizes. Please watch the full versions of them with this demo including additional scenes.
Our code is implemented using PyTorch1.7, and was tested under the following setting:
Caution: since there is "align_corners=True" option in "nn.functional.interpolate" and "nn.functional.grid_sample" in PyTorch1.7, we recommend you to follow our settings. Especially, if you use the other PyTorch versions, it may lead to yield a different performance.
DeMFI
└── Datasets
├──── Adobe_240fps_blur
├──── test
├──── 720p_240fps_1
├──── 00001.png
├──── ...
└──── 00742.png
...
├──── IMG_0183
├──── test_blur
├──── ...
├──── train
├──── ...
├──── train_blur
├──── ...
DeMFI
└── checkpoint_dir
└── DeMFInet_exp1
├── DeMFInet_exp1_latest.pt
# For evaluating on Adobe240
python main.py --gpu 0 --phase 'test' --exp_num 1 --test_data_path './Datasets/Adobe_240fps_blur' --N_tst 3 --multiple_MFI 8
# For evaluating on GoPro(HD)
python main.py --gpu 0 --phase 'test' --exp_num 1 --test_data_path './Datasets/GoPro_blur' --N_tst 3 --multiple_MFI 8
First prepare your own blurry videos in '.png' format in \<source_path>/custom_path by following a hierarchy as belows:
DeMFI
└── custom_path
├── scene1
├── 'xxx.png'
├── ...
└── 'xxx.png'
...
├── sceneN
├── 'xxxxx.png'
├── ...
└── 'xxxxx.png'
3. Since DeMFI-Net takes **4 input frames**, each scene must have at least 4 frames.
4. Download the pre-trained weight of DeMFI-Net<sub>*rb*</sub>(5,N<sub>*tst*</sub>), which was trained by Adobe240, from [this link](https://www.dropbox.com/s/rooyxrlv71ewnf5/DeMFInet_exp1_latest.pt?dl=0) to place in **\<source_path\>/checkpoint_dir/DeMFInet_exp1**.
DeMFI └── checkpoint_dir └── DeMFInet_exp1 ├── DeMFInet_exp1_latest.pt
5. Run **main.py** with the following options in parse_args (ex) joint Deblurring and Multi-Frame Interpolation (x8)):
```bash
python main.py --gpu 0 --phase 'test_custom' --exp_num 1 --N_tst 3 --multiple_MFI 8 --custom_path './custom_path' --patch_boundary 32
DeMFI
└── Datasets
├──── Adobe_240fps_blur
├──── test
├──── 720p_240fps_1
├──── 00001.png
├──── ...
└──── 00742.png
...
├──── IMG_0183
├──── test_blur
├──── ...
├──── train
├──── ...
├──── train_blur
├──── ...
python main.py --phase 'train' --exp_num 1 --train_data_path './Datasets/Adobe_240fps_blur' --test_data_path './Datasets/Adobe_240fps_blur' --N_trn 5 --N_tst 3
If you want to visualze several components, you can add '--visualization_flag True' visualization_flag option to get a 'visualizations' folder.
The diverse components are basically computed by min-max normalization as here, so you can freely modify them to any other visualization technique.
The below image is an example when t=7/8 for a reference.
If you have any question, please send an email to [Jihyong Oh] - jhoh94@kaist.ac.kr.
The source codes including the checkpoint can be freely used for research and education only. Any commercial use should get formal permission first.
We follow a blurry formation setting from BIN (Blurry Video Frame Interpolation) by averaging 11 consecutive frames at a stride of 8 frames over time to synthesize blurry frames captured by a long exposure, which finally generates blurry frames of 30fps with K = 8 and τ = 5 in Eq. 1. We thank the authors for sharing codes for their awesome works.