Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention

Official Implementation of "Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention" ( Paper )

Setup

For simplicity, you can directly run bash install.sh, which includes the following steps:

0. install pytorch 1.9.1 and other dependencies, e.g.,

   
   pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html # this might need to be changed due to cuda driver version 

pip install -r requirements.txt

1. install GroundingDINO and download pre-trained weights

cd GroundingDINO && python3 setup.py install

mkdir $PWD/GroundingDINO/weights/

wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth -O $PWD/GroundingDINO/weights/groundingdino_swint_ogc.pth

wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth -O $PWD/GroundingDINO/weights/groundingdino_swinb_cogcoor.pth

## Dataset 

• VG150
• COCO

  
  prepare the dataset under the folder ```data``` with the [instruction](datasets/data.md)

Closed-set SGG

For training OvSGTR (w. Swin-T) on VG150, running with this command

bash scripts/DINO_train_dist.sh vg ./config/GroundingDINO_SwinT_OGC_full.py  ./data  ./logs/ovsgtr_vg_swint_full ./GroundingDINO/weights/groundingdino_swint_ogc.pth

or

bash scripts/DINO_train_dist.sh vg ./config/GroundingDINO_SwinB_full.py  ./data  ./logs/ovsgtr_vg_swinb_full ./GroundingDINO/weights/groundingdino_swinb_cogcoor.pth 

for using Swin-B backbone. you might need to change the default devices of CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 in the script. Notice that the actual batch size = batch size (default 4 in config files) * num gpus. For inference, running with this command

bash scripts/DINO_eval.sh vg [config file] [data path] [output path] [checkpoint]

or

bash scripts/DINO_eval_dist.sh vg [config file] [data path] [output path] [checkpoint]

with multiple GPUs (there is a slight difference of the result output by DINO_eval.sh and DINO_eval_dist.sh due to data dividing and gathering).

Checkpoints

OvD-SGG

for OvD-SGG mode, set sg_ovd_mode = True in the config file (e.g., config/GroundingDINO_SwinT_OGC_ovd.py). Following "Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning" and VS3, we split the VG150 into two parts, i.e., base objects VG150_BASE_OBJ_CATEGORIES, and novel objects in VG150_NOVEL2BASE. For PREDCLS, please set use_gt_box=True when calling inference scripts.

Checkpoints

OvR-SGG

for OvR-SGG mode, set sg_ovr_mode = True in the config file (e.g., config/GroundingDINO_SwinT_OGC_ovr.py). Base object categories VG150_BASE_PREDICATE and novel object categories VG150_NOVEL_PREDICATE can be found in the datasets/vg.py.

Checkpoints

OvD+R-SGG

For OvD+R-SGG mode, set both sg_ovd_mode = True and sg_ovr_mode = True (e.g., config/GroundingDINO_SwinT_OGC_ovdr.py)

Checkpoints

Acknowledgement

Thank Scene-Graph-Benchmark.pytorch and GroundingDINO for their awesome code and models.

Citation

Please cite OvSGTR in your publications if it helps your research:

@inproceedings{chen2024expanding,
  title={Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention},
  author={Chen, Zuyao and Wu, Jinlin and Lei, Zhen and Zhang, Zhaoxiang and Chen, Changwen},
  booktitle={European Conference on Computer Vision},
  year={2024}
}