REPP is a learning based post-processing method to improve video object detections from any object detector. REPP links detections accross frames by evaluating their similarity and refines their classification and location to suppress false positives and recover misdetections.
REPP improves video detections both for specific Image and Video Object Detectors and it supposes a light computation overhead.
REPP has been tested with Python 3.6.
Its dependencies can be found in _repprequirements.txt file.
pip install -r repp_requirements.txt
Video detections must be stored with pickle as tuples (video_name, {frame_dets}) as following:
("video_name", {"000001": [ det_1, det_2, ..., det_N ],
"000002": [ det_1, det_2, ..., det_M ]},
...)
If the stored predictions file contains detections for different videos, they must be saved as a stream of tuples with the above format.
And each detection must have the following format:
det_1: {'image_id': image_id, # Same as the used in ILSVRC if applies
'bbox': [ x_min, y_min, width, height ],
'scores': scores, # Vector of class confidence scores
'bbox_center': (x,y) } # Relative bounding box center
_bboxcenter coordinates are bounded by 0 and 1 and referes to the center of the detection when the image has been padded vertically or horizontally to fit a square shape.
Check this code for a better insight about the predictions format.
Post-processed detections can be saved both with the COCO or IMDB format.
python REPP.py --repp_cfg ./REPP_cfg/cfg.json --predictions_file predictions_file.pckl --store_coco --store_imdb
As a REPP configuration file, you can use either _fgfa_reppcfg.json or _yolo_reppcfg.json. The first one works better with high performing detectors such as SELSA or FGFA and the second one works better for lower quality detectors. We recommend to set _appearancematching to false in the config file since it requires a non-trivial training of extra models and it's not mandatory for the performance bossting. If needed, the following config parameters can be tunned:
Below you will find instructions to perform any video predictions with YOLOv3 and apply REPP.
In order to reproduce the results of the paper, you can download the predictions of the different models from the following link and locate them in the project folder as structured in the downloaded zip folder.
Imagenet VID dataset must be downloaded and stored with the following folder structure:
/path/to/dataset/ILSVRC2015/
/path/to/dataset/ILSVRC2015/Annotations/DET
/path/to/dataset/ILSVRC2015/Annotations/VID
/path/to/dataset/ILSVRC2015/Data/DET
/path/to/dataset/ILSVRC2015/Data/VID
/path/to/dataset/ILSVRC2015/ImageSets
Following commands will apply the REPP post-processing and will evaluate the results by calculating the mean Average Precision for different object motions:
# YOLO
python REPP.py --repp_cfg ./REPP_cfg/yolo_repp_cfg.json --predictions_file './demos/YOLOv3/predictions/base_preds.pckl' --evaluate --annotations_filename ./data_annotations/annotations_val_ILSVRC.txt --path_dataset /path/to/dataset/ILSVRC2015/ --store_coco --store_imdb
> {'mAP_total': 0.7506216640807263, 'mAP_slow': 0.825347229618856, 'mAP_medium': 0.742908326433008, 'mAP_fast': 0.5657881762511975}
# FGFA
python REPP.py --repp_cfg ./REPP_cfg/fgfa_repp_cfg.json --predictions_file './demos/Flow-Guided-Feature-Aggregation/predictions/base_preds.pckl' --evaluate --annotations_filename ./data_annotations/annotations_val_ILSVRC.txt --path_dataset /path/to/dataset/ILSVRC2015/ --store_coco --store_imdb
> {'mAP_total': 0.8009014265948871, 'mAP_slow': 0.8741923949671497, 'mAP_medium': 0.7909183123072739, 'mAP_fast': 0.6137783055850773}
# SELSA
python REPP.py --repp_cfg ./REPP_cfg/selsa_repp_cfg.json --predictions_file './demos/Sequence-Level-Semantics-Aggregation/predictions/old_preds.pckl' --evaluate --annotations_filename ./data_annotations/annotations_val_ILSVRC.txt --path_dataset /path/to/dataset/ILSVRC2015/ --store_coco --store_imdb
> {'mAP_total': 0.8421329795837483, 'mAP_slow': 0.8871784038276325, 'mAP_medium': 0.8332090469178383, 'mAP_fast': 0.7109387713303483}
Instead of download the base predictions, you can also compute them. To do so, you must install the proper dependencies for each model as specified in the original model repositories (YOLOv3, FGFA, SELSA). You must also download their weights and config files from the following link and locate them in the project folder as structured in the downloaded zip file. Then execute the following commands:
# YOLO
cd demos/YOLOv3/
python get_repp_predictions.py --yolo_path ./pretrained_models/ILSVRC/1203_1758_model_8/ --repp_format --add_appearance --from_annotations ../../data_annotations/annotations_val_ILSVRC.txt --dataset_path /path/to/dataset/ILSVRC2015/Data/VID/
# FGFA
cd demos/Flow-Guided-Feature-Aggregation/fgfa_rfcn/
python get_repp_predictions.py --det_path 'path_to_dataset/ILSVRC2015/'
# SELSA
cd demos/Sequence-Level-Semantics-Aggregation/
python experiments/selsa/get_repp_predictions.py --dataset_path 'path_to_dataset/ILSVRC2015/'
REPP can be also applied to the predictions from any video as long as they have the specified REPP format. Following code shows how to compute YOLO predictions from any video and apply REPP post-processing.
# Extract YOLOv3 predictions
cd demos/YOLOv3/
python get_repp_predictions.py --yolo_path ./pretrained_models/ILSVRC/1203_1758_model_8/ --repp_format --add_appearance --from_video ./test_images/video_1.mp4
# Apply REPP
cd ../..
python REPP.py --repp_cfg ./REPP_cfg/yolo_repp_cfg.json --predictions_file './demos/YOLOv3/predictions/preds_repp_app_video_1.pckl' --store_coco
The present project includes trained linking models both to perform the detection matching with and without appearance descriptors. These models have been trained with data from Imagenet VID, but they are able to improve detections for any other dataset or custom video. These Logistic Regression models have been trained using the following steps, that can be adapted to any other custom dataset:
python create_triplet_ilsvrc_annotations.py --path_dataset '/path/to/dataset/ILSVRC2015/'
python clf_dataset_generation.py --path_dataset '/path/to/dataset/ILSVRC2015/' --add_appearance
python train_clf_model.py --add_appearance
Previous steps include appearance features calculated from a pretrained YOLOv3 model. If you are going to use a different dataset or detection model, it's recommended to omit the _--addappearance parameter.
@inproceedings{sabater2020repp,
title={Robust and efficient post-processing for Video Object Detection},
author={Alberto Sabater, Luis Montesano, Ana C. Murillo},
booktitle={International Conference of Intelligent Robots and Systems (IROS)},
year={2020}
}