Deep-Rep-MFIR

Official implementation of Deep Reparametrization of Multi-Frame Super-Resolution and Denoising

Publication: Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. ICCV 2021 oral [Arxiv]

Note: The code for our CVPR2021 paper "Deep Burst Super-Resolution" is available at goutamgmb/deep-burst-sr

Overview

We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image, which transforms the MAP objective to a deep feature space. The deep reparametrization allows us to directly model the image formation process in the latent space, and to integrate learned image priors into the prediction. Our approach thereby leverages the advantages of deep learning, while also benefiting from the principled multi-frame fusion provided by the classical MAP formulation. We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets. Our approach sets a new state-of-the-art for both tasks, demonstrating the generality and effectiveness of the proposed formulation.

[Classical multi-frame image restoration approaches minimize a reconstruction error between the observed images and the simulated images to obtain the output image y. In contrast, we employ an encoder E to compute the reconstruction error in a learned feature space. The reconstruction error is minimized w.r.t. a latent representation z, which is then passed through the decoder D to obtain the prediction y.]

Table of Contents

• Installation
• Toolkit Overview
• Datasets
• Evaluation
• Model Zoo
• Training
• Acknowledgement

Installation

Clone the Git repository.

git clone https://github.com/goutamgmb/deep-rep.git

Install dependencies

Run the installation script to install all the dependencies. You need to provide the conda install path (e.g. ~/anaconda3) and the name for the created conda environment (here env-deeprep).

bash install.sh conda_install_path env-deeprep

This script will also download the default DeepRep networks and create default environment settings.

Update environment settings

The environment setting file admin/local.py contains the paths for pre-trained networks, datasets etc. Update the paths in local.py according to your local environment.

Toolkit Overview

The toolkit consists of the following sub-modules.

• actors: Contains the actor classes for different trainings. The actor class is responsible for passing the input data through the network can calculating losses.
• admin: Includes functions for loading networks, tensorboard etc. and also contains environment settings.
• data: Contains functions for generating synthetic bursts, camera pipeline, processing data (e.g. loading images, data augmentations).
• data_specs: Information about train/val splits of different datasets.
• dataset: Contains integration of datasets such as BurstSR, SyntheticBurst, ZurichRAW2RGB, OpenImages, Grayscale denoising and Color denoising.
• evaluation: Scripts to run and evaluate models on standard datasets.
• external: External dependencies, e.g. PWCNet.
• models: Contains different layers and network definitions.
• train_settings: Default training settings for different models.
• trainers: The main class which runs the training.
• util_scripts: Util scripts to e.g. download datasets.
• utils: General utility functions for e.g. plotting, data type conversions, loading networks.

Datasets

The toolkit provides integration for following datasets which can be used to train/evaluate the models.

Evaluation

You can run the trained models on the included datasets and compute the quality of predictions using the evaluation module.

Note: Please prepare the necessary datasets as explained in Datasets section before running the models.

Model Zoo

Here, we provide pre-trained network weights and report their performance.

Note: The models have been retrained using the cleaned up code, and thus can have small performance differences compared to the models used for the paper.

Training

You can train the models using the run_training.py script. Please download and set up the necessary datasets as described in Datasets section, before starting the trainings. You will also need a pre-trained PWC-Net model to start the trainings. The model is automatically downloaded from the install.sh script. You can also download it manually using

gdown https://drive.google.com/uc\?id\=1s11Ud1UMipk2AbZZAypLPRpnXOS9Y1KO -O pretrained_networks/pwcnet-network-default.pth

You can train a model using the following command

python run_training.py MODULE_NAME PARAM_NAME

Here, MODULE_NAME is the name of the training module (e.g. deeprep), while PARAM_NAME is the name of the parameter setting file (e.g. sr_synthetic_default). We provide the default training settings used to obtain the results in the ICCV paper.

• sr_synthetic_default: Train RAW burst super-resolution model on synthetic bursts.
• sr_burstsr_default: Train RAW burst super-resolution model on real-world BurstSR.
• denoise_grayscale_pwcnet: Train grayscale burst denoising model using pre-trained PWC-Net.
• denoise_color_pwcnet: Train color burst denoising model using pre-trained PWC-Net.
• denoise_grayscale_customflow: Train grayscale burst denoising model using a custom optical flow network.
• denoise_color_customflow: Train color burst denoising model using a custom optical flow network.

Acknowledgement

The toolkit uses code from the following projects:

• Correlation module from ClementPinard/Pytorch-Correlation-extension.
• Forward and inverse camera pipeline code from timothybrooks/unprocessing.
• PWC-Net code from sniklaus/pytorch-pwc
• PWC-Net pre-trained models from NVlabs/PWC-Net
• MS-SSIM code from jorge-pessoa/pytorch-msssim.
• Core code for the toolkit (e.g. saving and loading models) from visionml/pytracking