How to generate a graph from an image?

[LeonLee8Hang]

If a scene graph need to be generated by RelTR, what should be done? Since this command only detect the objects in the image: python inference.py --img_path ./demo/vgx.jpg --resume ./ckpt/checkpoint0149.pth

So, please kindly help to tell me how to do, thanks very much!

[yrcong]

Hi, the outputs of the inference should be a list of triplets including subjects, objects and their relations. Not only entities. These triplets make up the scene graph.

Do you mean the automatic visualization? or just get the semantic results? If you want to visualize the scene graph, Graphviz is a good tool for you.

[LeonLee8Hang]

Hi, thanks for your response.

It seems the work : RelTR is excellent. I am trying to using it to generate a scene graph, which is similar to the "demo.png". I am a freshman in this field, so I wonder whether there is scripts of generating a scene graph as the demo image which could be shared? That would be wonderful.

Thanks!

[yrcong]

You could use Graphviz if you want to draw scene graphs automatically.

But there is no tool that can draw scene graphs like the figures in papers. They are all hand drawn.

[LeonLee8Hang]

Hi, I still confuse about the scene graph generation, the question is how to use the outputs data to construct a Scene Graph? when execute the command to run the inference, outputs of the model will be generated as below:

python inference.py --img_path ./demo/vg3.jpg --resume ./ckpt/checkpoint0149.pth

propagate through the model

outputs = model(img)

outputs:

{'pred_logits': tensor([[[-14.0701, -6.9378, -3.6805, ..., -2.2955, -8.1152, 9.3173], ... [-11.5406, -4.8710, -5.6274, ..., -5.3334, -11.0269, 8.4589]]]), 'pred_boxes': tensor([[[0.9285, 0.2640, 0.1396, 0.3936], ... [0.5625, 0.2458, 0.1401, 0.2452]]]), 'sub_logits': tensor([[[-10.0269, -5.9606, -2.0709, ..., -2.8782, -11.7878, 6.2765], ... [-14.5083, -7.4713, -5.7031, ..., -5.1850, -9.3440, 8.3625]]]), 'sub_boxes': tensor([[[0.4524, 0.2413, 0.4110, 0.3153], ... [0.2999, 0.2770, 0.6007, 0.5570]]]), 'obj_logits': tensor([[[-11.4217, -6.2552, -5.2713, ..., -3.9685, -14.2370, 6.4583], ... [-11.3257, -3.0029, -1.1787, ..., -3.2346, -1.6754, 7.0650]]]), 'obj_boxes': tensor([[[0.4553, 0.2449, 0.4414, 0.3307], ...

Thanks.

[LeonLee8Hang]

Hi, could you please kindly help to let me know how to use the outputs data generated by the model to create a Scene Graph? Thanks

[LeonLee8Hang]

Hi, I still have the same question of how to create a Scene Graph from output data, Since it is urgent, could you share the function script?

In the output data, how to know the relationship between subject and object?

Output dict with the following elements:

• "pred_logits",
• "pred_boxes",
• "sub_logits",
• "obj_logits",
• "sub_boxes",
• "obj_boxes",
• "aux_outputs"

Thanks.

[yrcong]

"pred_logits": the entity class probability (this is not related to SGG) "pred_boxes": the entity bounding box (this is not related to SGG) "sub_logits": N subject class probability
"obj_logits": N object class probability "sub_boxes": N subject bounding box "obj_boxes": N object bounding box "rel_logits": N * relationship class probability "aux_outputs": output for auxiliary loss (this is not related to SGG)

For example, the information of the k-th triplet proposal should be: output["sub_logits"][k], output["sub_boxes"][k], output["obj_logits"][k], output["obj_boxes"][k], output["rel_logits"][k]

As you see in the inference.py, you can filter some proposals when the confidence scores (subject/object/relationship) are low.

[LeonLee8Hang]

Hi, thanks for your response very much. Here following one question:

i.e. : the data of the value: output["sub_logits"][k] is numeral value, such as -11.7209, how to associate it with VG classes, like 'airplane'? Does there any formula or function doing the transformation?

Thanks

[yrcong]

Something is wrong...

output["sub_logits"] should be a tensor with the shape [query_number, class_num+1]! So output["sub_logits"][k] should be a 1- d tensor. The shape should be [152] (or if 151 I forgot if there is a padding class). This is the probability corresponding to the 151 VG entity classes (with "background").

I think inference.py is a good demo. You see the output figure has shown the triplets. It's better to read it line by line. I think it is easy to understand the output structure.

[LeonLee8Hang]

Thanks very much!