Contrastive Learning for Weakly Supervised Phrase Grounding

By Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem

(ECCV 2020 Spotlight)

Available on Arxiv: https://arxiv.org/abs/2006.09920

Project Page: http://tanmaygupta.info/info-ground/

BibTex:

@article{gupta2020contrastive,
  title={Contrastive Learning for Weakly Supervised Phrase Grounding},
  author={Gupta, Tanmay and Vahdat, Arash and Chechik, Gal and Yang, Xiaodong and Kautz, Jan and Hoiem, Derek},
  booktitle={ECCV},
  year={2020}
}

Requirements

Create a conda environment with all dependencies provided in the environment.yml file using

conda env create -f environment.yml

Activate the environment with

conda activate info-ground

All commands in the following sections are to be executed in the same directory as this README.md file.

Setup file paths and data

Get object detections

We provide detections for COCO and Flickr30K images computed using a FasterRCNN model trained on VisualGenome object and attribute annotations originally used in the Bottom-Up and Top-Down Attention work and then reused in a recent weakly supervised phrase grounding work Align2Ground that we compare to.

We use a lightly modified fork of the pytorch implementation available here to extract bounding boxes, scores, and features from a set of images and save them in hdf5 format.

Download and extract detections to a desired location:

• COCO [20 GB]
• Flickr30K [5 GB]

Update det_dir in yaml/coco.yml or yaml/flickr.yml to location where the detections were extracted.

Construct context-preserving negative captions

Follow the instructions for whichever dataset you want to train on.

Learn to ground

Once we have the following, we are ready to train our grounding model:

• Detections on train and val sets for the dataset you want to train on (COCO or Flickr30K)
• Negatives with cached features for the train and val set for the same dataset

Evaluate on Flickr

To evaluate on Flickr, follow the instructions above to setup Flickr file paths, download/extract the dataset, and download object detections. If needed also run setup_flickr.sh to copy files from NFS to local disk after modifying NFS_DATA and LOCAL_DATA paths in the script.