LIGHTEN-Learning-Interactions-with-Graphs-and-hierarchical-TEmporal-Networks-for-HOI

Introduction

This repository contains code for LIGHTEN HOI detection pipeline, proposed in the ACM MM'20 paper: [LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos]().

Installation

LIGHTEN is implemented in Pytorch1.4 with CUDA-10.1 in python3.8. Other python packages can be installed using :

pip install -r requirements.txt

Setting up the codebase

Datasets

• Download RGB frames for CAD120 videos from CAD120 dataset page
• Download COCO image directory (traintest2017) from COCO website

  Pre-trained models

  Download the pretrained models from the following folders : i) CAD120 checkpoints ii) V-COCO checkpoints

  Configuration

  Set the corresponding paths to data and pre-trained models in config.py file. Hyper-paramters and model configurations can be set from this file. The directory structure after setting up looks like :

  LIGHTEN-Learning-Interactions-with-Graphs-and-hierarchical-TEmporal-Networks-for-HOI/
  CAD120/
  checkpoints/
    checkpoint_GCN_frame_detection.pth
    checkpoint_GCN_segment_detection.pth
  data/
    training_data.p
    testing_data.p
  models/
  V-COCO/
  checkpoints/
  data/
    training_data.p
    testing_data.p
    action_index.json
    Test_Faster_RCNN_R-50-PFN_2x_VCOCO_Keypoints.pkl
  models/

Running the code

CAD120

• Resnet frame-wise features can be precomputed and stored beforehand as :

  cd CAD120/
  python compute_RoI_feats.py

  • This will create two new files at CAD120/data/, which contain image features from backbone module. Alternately, the precomputed feature files can be downloaded from this google drive folder.
• Training and Testing the LIGHTEN model for CAD120 can be done as follows :

  cd CAD120/
  python train_CAD120.py
  python test_CAD120.py

  V-COCO

• Resnet image features can be precomputed and stored using :

  cd V-COCO/
  python compute_RoI_feats.py

  • This will store resnet features in the directory : V-COCO/data/
• Training, Validation and Testing of LIGHTEN on V-COCO (image-setting) can be done as :

  cd V-COCO/
  python train_VCOCO.py
  python eval_VCOCO.py
  python test_VCOCO.py

• Note that eval_VCOCO.py evaluates only action label detection, and uses ground-truth object detections similar to train_VCOCO.py. However, test_VCOCO.py evaluates the model using faster-RCNN-FPN object detections, and computes the final mAP score as per the evaluation script available at VSRL Repository

Citation