Alessandro Flaborea, Luca Collorone, Guido D'Amely, Stefano D'Arrigo, Bardh Prenkaj, Fabio Galasso
The official PyTorch implementation of the IEEE/CVF International Conference on Computer Vision (ICCV) '23 paper Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection.
conda env create -f environment.yaml
conda activate mocodad
You can download the extracted poses for the datasets HR-Avenue, HR-ShanghaiTech and HR-UBnormal from the GDRive.
Place the extracted folder in a ./data
folder and change the configs accordingly.
To adapt your custom dataset to work with MoCoDAD, you can follow the structure below or look at the UBnormal dataset description here.
We also provide the code to extract poses and track actors in videos in the _annotations
folder.
{your_custom_dataset}
|
|__________ training
| |
| |__________ trajectories
| |
| |_________{scene_id}_{clip_id}
| |
| |_________00001.csv
| |_________...
| |_________0000{n}.csv
|
|__________ testing
| |
| |__________ trajectories
| | |
| | |_________{scene_id}_{clip_id}
| | |
| | |_________00001.csv
| | |_________...
| | |_________0000{n}.csv
| |
| |__________ test_frame_mask
| |
| |_______________{scene_id}_{clip_id}.npy
| |_______________...
| |_______________{scene_id}_{clip_id}.npy
|
|__________ validating
|
|__________ trajectories
| |
| |_________{scene_id}_{clip_id}
| |
| |_________00001.csv
| |_________...
| |_________0000{n}.csv
|
|__________ test_frame_mask
|
|_______________{scene_id}_{clip_id}.npy
|_______________...
|_______________{scene_id}_{clip_id}.npy
To train MoCoDAD, you can select the different type of conditioning of the model. The default parameters achieve the best results reported in the paper
In each config file you can choose the conditioning strategy and change the diffusion process parameters:
conditioning_strategy
Diffusion Process
Update the args 'data_dir', 'test_path', 'dataset_path_to_robust' with the path where you stored the datasets. To better track your experiments, change 'dir_name' and the wandb parameters.
To train MoCoDAD:
python train_MoCoDAD.py --config config/[Avenue/UBnormal/STC]/{config_name}.yaml
The training config is saved the associated experiment directory (/args.exp_dir/args.dataset_choice/args.dir_name
).
To evaluate the model on the test set, you need to change the following parameters in the config:
Test MoCoDAD
python eval_MoCoDAD.py --config /args.exp_dir/args.dataset_choice/args.dir_name/config.yaml
additional flag you can use:
The checkpoints for the pretrained models on the three datasets can be found HERE. To evaluate them follow the following steps:
/checkpoints/[Avenue/UBnormal/STC]/pretrained_model
python eval_MoCoDAD.py --config `/checkpoints/[Avenue/UBnormal/STC]/pretrained_model/mocodad_test.yaml]
We provide the code to visualize frames, poses and anomaly scores. Follow the instruction in visualize for further details.
@InProceedings{Flaborea_2023_ICCV,
author = {Flaborea, Alessandro and Collorone, Luca and di Melendugno, Guido Maria D'Amely and D'Arrigo, Stefano and Prenkaj, Bardh and Galasso, Fabio},
title = {Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {10318-10329}
}