From Predictions to Scene Graph

[BlueBell93]

Hello, thank you for this great work and the provided resources.

I have a question regarding how to get a scene graph from the models predictions.

I am using the config file configs/config_3DSSG_full_l160.yaml and your trained model to infer predictions on one sample of the dataset.

For the node class prediction I applied a softmax function to the node class logits and used the index of the highest value for each "node" to determine its class. For the edge class predictions I applied the sigmoid function and used a threshold of 0.5 to determine the multi-relationship predictions.

I am uncertain about how to proceed from these predictions to actually generate the scene graph, specifically how to know which nodes are connected based on the edge class predictions.

• How should the predicted edges be interpreted to connect the nodes?
• Are there any utilities or functions in the codebase that I might have missed that facilitate this process?

[ShunChengWu]

I don't have any files with me, so I will reply with my memory. If I remember correctly, the output file contains a graph file that has a mapping from the edge index to the actual node index. This mapping is generated in the data loader, which you can find here: https://github.com/ShunChengWu/3DSSG/blob/4b783ecdc6caba1515b361f8a0643d0c2d568f52/ssg/dataset/dataloader_SGFN.py#L333-L334 The edge_indices_3D is the edges to be predicted and the input oid2idx is the mapping of the input index to the real object index.