SGFormer
Official implementation of "SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generation", AAAI 2024
Figure, Overview pipeline of our proposed SGFormer. We leverage PointNet to initialize the node and edge features in the 3D scene and use LLMs ( i.e., ChatGPT) to enrich the object description text of the dataset as semantic knowledge. The SGFormer main consists of two carefully designed components: a Graph Embedding Layer and a Semantic Injection Layer.
Dataset
We followed the dataset processing method of EdgeGCN A quick glance at some features of our cleaned 3DSSG-O27R16 dataset (compared to the original 3DSSG dataset):
- dense point cloud representation with color and normal vector info. encoded - see Sec. A - Point Cloud Sampling;
- with same scene-level split applied on 3DSSG - but with FullScenes (i.e., original graphs) instead of SubScenes (subgraphs of 4-9 nodes in 3DSSG);
- with small / partial scenes of low quality excluded - see this list (officially announced in 3DSSG's FAQ Page);
- with object-level class imbalance alleviated - see Sec. B1 - Node (object) Remapping;
- with edge-wise comparative relationships (e.g.,
more-comfortable-than) filtered out - we focus on structural relationships instead; - reformulate the edge predictions from a multi-label classification problem to a multi-class one - see Sec. B2 - Edge (Relationship) Relabelling;
To obtain the 3DSSG-O27R16 dataset, please follow the instructions in EdgeGCN's project page.
Code
- [x] SGFormer.py: CoreNetwork of SGformer.
Acknowledgement
This resipotry is based on EdgeGCN
Citation
If you find our data or project useful in your research, please cite:
@inproceedings{lv2024sgformer,
title={SGFormer: Semantic Graph Transformer for Point Cloud-Based 3D Scene Graph Generation},
author={Lv, Changsheng and Qi, Mengshi and Li, Xia and Yang, Zhengyuan and Ma, Huadong},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={38},
number={5},
pages={4035--4043},
year={2024}
}