search

Wangt-CN / VC-R-CNN

[CVPR 2020] The official pytorch implementation of ``Visual Commonsense R-CNN''
MIT License
352 stars 61 forks source link

readme

Visual Commonsense R-CNN (VC R-CNN)

LICENSE Python PyTorch

[NEW]: We have provided the training code of VC R-CNN and detailed readme file. :star2:

[NEW]: the VC Feature pretrained on MSCOCO is provided. Just have a try! :star2:

This repository contains the official PyTorch implementation and the proposed VC feature for CVPR 2020 Paper "Visual Commonsense R-CNN". For technical details, please refer to:

Visual Commonsense R-CNN
Tan Wang, Jianqiang Huang, Hanwang Zhang, Qianru Sun
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020
Key Words:  Causal Intervention;  Visual Common Sense;  Representation Learning
[Paper], [Zhihu Article], [15min Slides], [Video]

Bibtex

If you find our VC feature and code helpful, please kindly consider citing:

@inproceedings{wang2020visual,
  title={Visual commonsense r-cnn},
  author={Wang, Tan and Huang, Jianqiang and Zhang, Hanwang and Sun, Qianru},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10760--10770},
  year={2020}
}

Contents

  1. Overview
  2. VC Feature
  3. VC R-CNN Framework
  4. Our experience you may need

Overview

This project aims to build a visual commonsense representation learning framework based on the current object detection codebase with un-/self-supervised learning.

Highlights:

  1. VC Feature
    • Effective: Our visual commonsense representation encodes the``sense-making'' knowledge between object RoIs with causal intervention rather than just trivial correlation prediction. Compared to the previous widely used Up-Down Feature, our VC can be regarded as an effective supplementary knowledge that models the interaction between objects for the downstream tasks.
    • Easy to Use: As we introduced in our paper, the VC Feature is extracted by providing the RoI boxes coordinates. Then the VC Feature can be just concatenated on the previous visual object features (e.g., Up-Down Feature) and ready to roll. (Ps: the concatenation maybe too simple for some cases or tasks, users can try something else and welcome feedback.)
    • Easy to Expand: With a learned VC R-CNN framework, we can easily extract VC Features for any images and prepare them as an ``augmentation feature'' for the currently used representation conveniently.
  2. VC R-CNN
    • Fast, Memory-efficient, Multi-GPU: Our VC R-CNN framework is based on the well-known maskrcnn-benchmark from facebook. Therefore, our VC R-CNN just inherit all its advantages. (It's pity that the detectron2 had not been released when I am working on this project, however maskrcnn-benchmark can be a stable version.)
    • Support customized dataset: Users can easily add COCO-style datasets to train VC R-CNN on other images.

What can you get from this repo? [The Road Map]


VC Feature

For easy-to-use, here we directly provide the pretrained VC Features on the entire MSCOCO dataset based on the Up-Down feature's boxes in the below links (The link is updated). The features are stored in tar.gz format. The previous features can be found in OLD_FEATURE.

10-100 VC Features per image:

10-100 Updown Features per image:

For those who may have no access to the Up-Down feature, here we also provide the Updown feature here. Then you can directly use numpy.concatenate (The feature dimension is 3072 : 2048+1024):

10-100 Updown Boxes

For users can extract VC Features if they want, here we also provide the Updown feature box coordinates:

How to use after download

Downstream Vision & Language Tasks

Please check Downstream Tasks for more details:

Some tips for using in downstream tasks


VC R-CNN Framework

Installation

Please check INSTALL.md for installation instructions.

Perform Training on COCO Dataset

Prepare Training Data

  1. First, you need to download the COCO dataset and annotations. We assume that you save them in /path_to_COCO_dataset/
  2. Then you need modify the path in vc_rcnn/config/paths_catalog.py, containing the DATA_DIR and DATASETS path.

Training Parameters

Running

Most of the configuration files that we provide assume that we are running 2 images on each GPU with 8 GPUs. In order to be able to run it on fewer GPUs, there are a few possibilities:

1. Single GPU Training: Modify the cfg parameters. Here is an example:

python tools/train_net.py --config-file "configs/e2e_mask_rcnn_R_101_FPN_1x.yaml" --skip-test SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS "(480000, 640000)" TEST.IMS_PER_BATCH 1 MODEL.RPN.FPN_POST_NMS_TOP_N_TRAIN 2000

Ps: To running more images on one GPU, you can refer to the [maskrcnn-benchmark].

2. Multi-GPU training: The maskrcnn-benchmark directly support the multi-gpu training with torch.distributed.launch. You can run the command like (you need change $NGPUS to the num of GPU you use):

python -m torch.distributed.launch --nproc_per_node=$NGPUS tools/train_net.py --config-file "path/to/config/file.yaml" --skip-test MODEL.RPN.FPN_POST_NMS_TOP_N_TRAIN images_per_gpu x 1000

Notes:

Evaluation (Feature Extraction)

1. Using your own model

Since the goal of our VC R-CNN is to train the visual commonsense representations by self-supervised learning, we have no metrics for evaluation and we treat it as the feature extraction process.

Specifically, you can just run the following command to achieve the features.

python -m torch.distributed.launch --nproc_per_node=$NGPUS tools/test_net.py --config-file "path/to/config/file.yaml" TEST.IMS_PER_BATCH images_per_gpu x $GPUS

Please note that before running, you need to set the suitable path for BOUNDINGBOX_FILE and FEATURE_SAVE_PATH in default.py. (Recall that just given image and bounding box coordinate, our VC R-CNN can extract the VC Feature)

2. Using our pretrained VC model on COCO

Here we also provide our pretrained VC model. You can put it into the model dictionary and set the last_checkpoint with the absolute path of model_final.pth. Then run the command:

python -m torch.distributed.launch --nproc_per_node=$NGPUS tools/test_net.py --config-file "path/to/config/file.yaml" TEST.IMS_PER_BATCH images_per_gpu x $GPUS

Add your Customized Dataset

1. Training on customized dataset

For learning VC Feature on your own dataset, the crux is to make your own dataset COCO-style (can refer to the data format in detection task) and design the dataloader file, for example coco.py and openimages.py. Here we provide an example for reference.

from vc_rcnn.structures.bounding_box import BoxList

class MyDataset(object):
    def __init__(self, ...):
        # load the paths and image annotation files you will need in __getitem__

    def __getitem__(self, idx):
        # load the image as a PIL Image
        image = ...

        # load the bounding boxes as a list of list of boxes.
        boxes = [[0, 0, 10, 10], [10, 20, 50, 50]]
        # and labels
        labels = torch.tensor([10, 20])

        # create a BoxList from the boxes. Please pay attention to the box FORM (XYXY or XYWH or another)
        boxlist = BoxList(boxes, image.size, mode="xyxy")
        # add the labels to the boxlist
        boxlist.add_field("labels", labels)
        # Here you can also add many other characters to the boxlist in addition to the labels, for example `image_id', `category_id' and so on.

        if self.transforms:
            image, boxlist = self.transforms(image, boxlist)

        # return the image, the boxlist and the idx in your dataset
        return image, boxlist, idx

    def get_img_info(self, idx):
        # get img_height and img_width. This is used if
        # we want to split the batches according to the aspect ratio
        # of the image, as it can be more efficient than loading the
        # image from disk
        return {"height": img_height, "width": img_width}

Then, you need modify the following files:

2. Extracting features of customized dataset

Recall that with the trained VC R-CNN, we can directly extract VC Features given raw images and bounding box coordinates. Therefore, the method to design dataloader is similar to the above. The only difference is you may want to load box coordinates file for feature extraction and the labels, classes is unnecessary.

You can also refer to openimages.py and vcr.py.

3. Some Tips and Traps


Our experience you may need

Here we provide our experience (mainly the failure 2333) in training our VC R-CNN and using VC Feature. And we hope it can provide some help or possible ideas to the users to further develop this field :)

1. For Training VC R-CNN

2. For the VC Feature


Acknowledgement

I really appreciate Kaihua Tang, Yulei Niu, Xu Yang, Jiaxin Qi, Xinting Hu and Dong Zhang for their greatly helpful advice and lending me GPUs!

If you have any questions or concerns, please kindly email to Tan Wang.