This is the repo for a open project detecting human object interactions in real-time, see more detail on our Tech Report.
GPU: Titan, Titan Black, Titan X, K20, K40, K80, GTX
You should install matlab to validate the training result of HOI-RT. You should install cuda, opencv and cudnn. Then set the 1-3 line of Makefile:
GPU=1
CUDNN=1
OPENCV=1
cd detection && git clone --recursive git@github.com:lmingyin/HOI-RT.git
cd $HOI-RT && make -j8
After successful installation, now you can test HOI-RT.
cd $HOI-RT/
./darknet detector test cfg/vcoco.data cfg/yolo-vcoco608.cfg ../yolo-vcoco608_80000.weights data/kick.jpg
V-COCO dataset builds off MS COCO, please download MS-COCO images and annotations(coco 2014 is enough), make sure all which in a new folder coco, the downloaded extracted image folders like train2014, val2014, test2014 should in the new folder images which under coco, the downloaded extracted annotations like instances_train2014.json, instances_val2014.json should in the new folder annotations which under coco.
cd coco
git clone --recursive https://github.com/s-gupta/v-coco.git
cd v-coco
python script_pick_annotations.py coco-data/annotations
cd $VCOCO_DIR/coco/PythonAPI/ && make
cd $VCOCO_DIR && make
cd v-coco
python vcoco_show.py
More training data should be loaded from our dataset1 and our dataset2 , then combine the two folder to RelationDataset and put it under the folder detection.
cd coco && mkdir filelist
cd v-coco && python vcoco_label.py
Finally, in folder filelist will generate a file trainVCOCO.txt. And in folder coco will outputs a folder named labels which contain all training labels.
cd detection && python voc_relation_label.py
Finally, trainOurs.txt will be generated in current folder, and training labels will be generated in every action folder in RelationDataset.
Copy trainOurs.txt to the folder filelist. And then
cd filelist && cat trainVCOCO.txt trainOurs.txt > train.txt
Load the pretrained model, and put it in the detection folder
Before training, you should set cfg/vcoco.data
train = Your_Path/coco/filelist/train.txt
then use the following command to train the model
cd ROI-RT/
make clean && make -j8
./darknet detector train cfg/vcoco.data cfg/yolo-vcoco608.cfg ../darknet19_448.conv.23
cd v-coco
python vcoco_test_action.py
You will get a folder vcoco_action_valid. Put it in the detection folder.
cd HOI-RT/cfg/
open vcoco.data and set
valid = Your_Path/coco/filelist/vcoco_test.txt
eval =
and then validate the model
cd HOI-RT/
./darknet detector valid cfg/vcoco.data cfg/yolo-vcoco608.cfg backup/yolo-vcoco608_80000.weights
in the current folder a folder results will be generated, you should put it in the folder vcoco_action_valid, and then
cd HOI-RT/matlab
run the script validate_action.m, you will get the APagent for every action.
python vcoco_test_relation.py
A folder vcoco_relation_valid will be generated, and put it in the folder detection.
cd HOI-RT/cfg/
open vcoco.data and set
valid = Your_Path/coco/filelist/vcoco_test.txt
eval = relation
and then validate the model
cd HOI-RT/
./darknet detector valid cfg/vcoco.data cfg/yolo-vcoco608.cfg backup/yolo-vcoco608_80000.weights
in current folder a folder results will be generated, you should put it in the folder vcoco_relation_valid
cd HOI-RT/matlab
run the script validate_relation.m, you will get the AProle for every action.