Alignment-free HDR Deghosting with Semantics Consistent Transformer

Steven Tel , Zongwei Wu , Yulun Zhang $^\dagger$, Barthélémy Heyrman, Cédric Demonceaux, Radu Timofte, and Dominique Ginhac, "Alignment-free HDR Deghosting with Semantics Consistent Transformer", ICCV, 2023

[arXiv][supp]

Note that there are some corrections made to the conference proceedings to address issues with the production of our benchmark input. We have now updated Table 3 and Figure 6 to reflect these changes. Please refer to Manuscript, Supplementary and Web.

Abstract: High dynamic range (HDR) imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. The essence is to leverage the contextual information, including both dynamic and static semantics, for better image generation. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. However, there is no research on jointly leveraging the dynamic and static context in a simultaneous manner. To delve into this problem, we propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules in the network. The spatial attention aims to deal with the intra-image correlation to model the dynamic motion, while the channel attention enables the inter-image intertwining to enhance the semantic consistency across frames. Aside from this, we introduce a novel realistic HDR dataset with more variations in foreground objects, environmental factors, and larger motions. Extensive comparisons on both conventional datasets and ours validate the effectiveness of our method, achieving the best trade-off on the performance and the computational cost.

Our Dataset and Benchmark

Our HDR dataset can be found here

We retrain several SOTA counterparts from the official codes. The quantitative performances are as follows:

The benchmarking results are available at Google Drive. The HDR-VDP-2 and PU21 metrics from the Table can be computed using the provided script:

matlab utils/hdrvdp2_pu21/run/test.m

Train and Test

We introduce SCTNet, an end-to-end model for HDR deghosting.

To train our model, please follow the training and evaluation steps:

python train_sct.py.py
python test_sct.py.py

Make sure that you have changed the path to your dataset/ckpts in both files.

To use our dataset, just set the right data path in the training file.

Citation

If you find this repo useful, please consider citing:

@INPROCEEDINGS{tel2023alignment,
  title={Alignment-free HDR Deghosting with Semantics Consistent Transformer},
  author={Tel, Steven and Wu, Zongwei and Zhang, Yulun and Heyrman, Barth{\'e}l{\'e}my and Demonceaux, C{\'e}dric and Timofte, Radu and Ginhac, Dominique},
  booktitle={ICCV}, 
  year={2023},
}

Acknowledgments

This repository is heavily based on HDR-Transformer. Thanks to their great work!