Our paper SGTR: End-to-end Scene Graph Generation with Transformer has been accepted by CVPR 2022.
Check INSTALL.md for installation instructions.
Check DATASET.md for instructions of dataset preprocessing.
SGTR performance:
The methods implemented in our toolkit and reported results are given in Model Zoo.md
Our codebase is developed based on cvpods, you can refer to usage of cvpods and tutorial.
You can download the pretrained DETR on ResNet-101 we used in the paper:
Unzip the checkpoint into the folder
Then, you need to modify the pre-trained weight parameter MODEL.WEIGHTS
in config.py playground/experiment/path/config_xx.py
to the path of corresponding pre-trained detector weight to make sure you load the detection weight parameter correctly.
Besides, you can also train your own detector weight by the provide configs in Model Zoo.md
You can run training by:
# activate the environments
conda activate sgtr
cvpods_env
# move to experiment directory
cd playground/sgg/detr.res101.c5.one_stage_rel_tfmer
# link the config file
rm config.py; ln -s config_vg.py config.py
pods_train --num-gpus 4
Single Machine 4-GPU Run
pods_train --num-gpus 4
If you want check the machine with available gpus:
pods_train --num-gpus 4 --gpu-check
MODEL.TEST_WEIGHTS
in config.py
to the trained model weight you can directly eval the model on validation or test set.
config.json
) in same directory of model weights.
# visual genome dataset
pods_test --num-gpus 4 DATASETS.TEST "(vg_test,)"
pods_test --num-gpus 4 DATASETS.TEST "(vg_val,)"
pods_test --num-gpus 4 DATASETS.TEST "(oiv6_test,)" pods_test --num-gpus 4 DATASETS.TEST "(oiv6_val,)"
## Citations
If you find this project helps your research, please kindly consider citing our papers in your publications.
@InProceedings{Li_2022_CVPR, author = {Li, Rongjie and Zhang, Songyang and He, Xuming}, title = {SGTR: End-to-end Scene Graph Generation with Transformer}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {19486-19496} }
# Q&A
- Import ipdb in anywhere in your code will cause the multi-process initialization error, try pdb when you debug in multi-process mode.
## Acknowledgment
This repository borrows code from scene graph benchmarking framework developed by [KaihuaTang](https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch) and [Rongjie Li](https://github.com/SHTUPLUS/PySGG)