By Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari and Cordelia Schmid
MARS is a strategy to learn a stream that takes only RGB frames as input but leverages both appearance and motion information from them. This is achieved by training a network to minimize the loss between its features and the Flow stream, along with the cross entropy loss for recognition. For more details, please refer to our CVPR 2019 paper and our website.
We release the testing code along trained models.
@inproceedings{crasto2019mars,
title={{MARS: Motion-Augmented RGB Stream for Action Recognition}},
author={Crasto, Nieves and Weinzaepfel, Philippe and Alahari, Karteek and Schmid, Cordelia},
booktitle={CVPR},
year={2019}
}
Python3
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
ffmpeg version 3.2.4
OpenCV with GPU support (will not be providing support in compiling this part)
Directory tree
dataset/
HMDB51/
../(dirs of class names)
../(dirs of video names)
HMDB51_labels/
results/
test.txt
trained_models/
HMDB51/
../(.pth files)
The datsets and splits can be downloaded from
To extract only frames from videos
python utils1/extract_frames.py path_to_video_files path_to_extracted_frames start_class end_class
To extract optical flows + frames from videos
export OPENCV=path_where_opencv_is_installed
g++ -std=c++11 tvl1_videoframes.cpp -o tvl1_videoframes -I${OPENCV}include/opencv4/ -L${OPENCV}lib64 -lopencv_objdetect -lopencv_features2d -lopencv_imgproc -lopencv_highgui -lopencv_core -lopencv_imgcodecs -lopencv_cudaoptflow -lopencv_cudaarithm
python utils1/extract_frames_flows.py path_to_video_files path_to_extracted_flows_frames start_class end_class gpu_id
## Models
Trained models can be found here. The names of the models are in the form of
stream_dataset_frames.pth
RGB_Kinetics_16f.pth indicates --modality RGB --dataset Kinetics --sample_duration 16
For HMDB51 and UCF101, we have only provided trained models for the first split.
For RGB stream:
python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"
For Flow stream:
python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality Flow --sample_duration 16 --split 1 \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"
For single stream MARS:
python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB \
--resume_path1 "trained_models/HMDB51/MARS_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"
For two streams RGB+MARS:
python test_two_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--resume_path2 "trained_models/HMDB51/MARS_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"
For two streams RGB+Flow:
python test_two_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB_Flow --sample_duration 16 --split 1 \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--resume_path2 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51/HMDB51_frames/" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"
python train.py --dataset Kinetics --modality RGB --only_RGB \
--n_classes 400 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--result_path "results/"
python train.py --dataset HMDB51 --modality RGB --split 1 --only_RGB \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/RGB_Kinetics_16f.pth" \
--result_path "results/"
python train.py --dataset HMDB51 --modality RGB --split 1 --only_RGB \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/RGB_Kinetics_16f.pth" \
--resume_path1 "results/HMDB51/PreKin_HMDB51_1_RGB_train_batch32_sample112_clip16_nestFalse_damp0.9_weight_decay1e-05_manualseed1_modelresnext101_ftbeginidx4_varLR2.pth" \
--result_path "results/"
python train.py --dataset Kinetics --modality Flow \
--n_classes 400 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--result_path "results/"
python train.py --dataset HMDB51 --modality Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/"
python train.py --dataset HMDB51 --modality Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--resume_path1 "results/HMDB51/PreKin_HMDB51_1_Flow_train_batch32_sample112_clip16_nestFalse_damp0.9_weight_decay1e-05_manualseed1_modelresnext101_ftbeginidx4_varLR2.pth" \
--result_path "results/"
python MARS_train.py --dataset Kinetics --modality RGB_Flow \
--n_classes 400 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--resume_path1 "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/" --checkpoint 1
python MARS_train.py --dataset HMDB51 --modality RGB_Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--result_path "results/" --checkpoint 1
python MARS_train.py --dataset HMDB51 --modality RGB_Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--MARS_resume_path "results/HMDB51/MARS_HMDB51_1_train_batch16_sample112_clip16_lr0.001_nesterovFalse_manualseed1_modelresnext101_ftbeginidx4_layeravgpool_alpha50.0_1.pth" \
--result_path "results/" --checkpoint 1
python MERS_train.py --dataset Kinetics --modality RGB_Flow \
--n_classes 400 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--resume_path1 "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/" --checkpoint 1
python MERS_train.py --dataset HMDB51 --modality RGB_Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MERS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--result_path "results/" --checkpoint 1
python MERS_train.py --dataset HMDB51 --modality RGB_Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--MARS_resume_path "results/HMDB51/MERS_HMDB51_1_train_batch16_sample112_clip16_lr0.001_nesterovFalse_manualseed1_modelresnext101_ftbeginidx4_layeravgpool_alpha50.0_1.pth" \
--result_path "results/" --checkpoint 1