EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer

Fei Wang, Dan Guo, Kun Li, Meng Wang. AAAI 2024.

Hefei University of Technology, Hefei Comprehensive National Science Center, Anhui Zhonghuitong Technology Co., Ltd.

[arXiv] | [PDF]

✒️:Abstract

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods.

📅: Update - Dec, 2023

Congratulations on our work being accepted by AAAI 2024!💥💥💥

Data preparation

• For train datasets from Oh et al. ECCV 2018, see the official repository here.

• For Real-world datatsets, we used three settings:

  • Static Dataset
  • Dynamic Dataset
  • Fabric Dataset from Davis et al. CVPR 2015 && TPAMI

• Real-world videos (or any self-prepared videos) need to be configured via the following:

  • Check the settings of val_dir in config.py and modify it if necessary.
  • To convert the Real-world video into frames: mkdir VIDEO_NAME && ffmpeg -i VIDEO_NAME.mp4 -f image2 VIDEO_NAME/%06d.png

  eg, mkdir ./val_baby && ffmpeg -i ./baby.avi -f image2 ./val_baby/%06d.png

  Tips: ffmpeg can also be installed by conda.

  • Modify the frames into frameA/frameB/frameC: python make_frameACB.py(remember adapt the 'if' at the beginning of the program to select videos.) Tips: Thanks to a fellow friend Peng Zheng for the help!

Env

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html einops

🌐 Usage

⚙ Network Architecture

Our EulerMormer is implemented in model/magnet.py.

• For Config: config.py
• For train: python main.py
• For test video: python test_video.py

Demo Baby Demo Drum Demo Cattoy

🔖:Citation

If you found this code useful please consider citing our paper:

@inproceedings{wang2024eulermormer,
  title={Eulermormer: Robust eulerian motion magnification via dynamic filtering within transformer},
  author={Wang, Fei and Guo, Dan and Li, Kun and Wang, Meng},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={6},
  pages={5345--5353},
  year={2024}
}