MomentDiff: Generative Video Moment Retrieval from Random to Real --NeurIPS 2023

by Pandeng Li¹, Chen-Wei Xie², Hongtao Xie¹, Liming Zhao², Lei Zhang¹, Yun Zheng² , Deli Zhao², Yongdong Zhang¹

¹ University of Science and Technology of China, ² Alibaba Group

Prerequisites

0. Clone this repo

1. Prepare datasets

Charades-STA : Download feature files for Charades-STA dataset.

VGG features and labels: Download Charades-STA-VGG,

SF+C features: We followed Moment-DETR to use Charades-STA-SF+C.

QVHighlights : Download official feature files for QVHighlights dataset from Moment-DETR.

SF+C features: Download moment_detr_features.tar.gz.

tar -xf path/to/moment_detr_features.tar.gz

TACoS : Prepare features for TACoS dataset.

C3D features: : According to VSLNet, convert the pre-trained C3D visual features from TALL.

ActivityNet : Prepare features for ActivityNet dataset.

C3D features: : According to VSLNet, convert the pre-trained C3D visual features from ActivityNet.

2. Install dependencies.

#使用conda python-3.7.16
conda create -n momentdiff python=3.7.16 
. activate
conda activate momentdiff
cd MomentDiff
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 torchtext==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt -i https://pypi.mirrors.ustc.edu.cn/simple
pip install tqdm ipython easydict tensorboard tabulate scikit-learn pandas -i https://pypi.mirrors.ustc.edu.cn/simple

Charades-STA

Training

Training with (VGG) and (SF+C) can be executed by running the shell below:

bash momentdiff/scripts/train_charades_vgg.sh 
bash momentdiff/scripts/train_charades_sf.sh 

Training on two anti-bias datasets can be executed by running the shell below:

bash momentdiff/scripts/train_anti_charades_len.sh 
bash momentdiff/scripts/train_anti_charades_mom.sh 

Training on Charades-CD and ActivityNet-CD can be executed by running the shell below:

bash momentdiff/scripts/train_charades_CD.sh 
bash momentdiff/scripts/train_anet_CD.sh 

LICENSE

The annotation files and many parts of the implementations are borrowed Moment-DETR and QD-DETR. Following, our codes are also under MIT license.