search

yrcong / RelTR

RelTR: Relation Transformer for Scene Graph Generation: https://arxiv.org/abs/2201.11460v2
248 stars 49 forks source link

What happens when there are no relations in a sample? #49

Open gzamps opened 6 months ago

gzamps commented 6 months ago

Hi and thanks for sharing this interesting work!

I am working on supporting another dataset in vg format. In my dataset I have images with no relation between the objects and I wonder how to make the network support this.

There is an issue since the images still contain boxes of objects but targets['rel_annotations'] is empty. Therefore in matcher forward pass, indices has a different length from indices1.

I'd appreciate your input. Thanks in advance!

yrcong commented 4 months ago

You can easily add a dummy predicate label "no_relation" and combine random bboxes with this dummy label.

gzamps commented 3 months ago

Hi and thanks for the response. I am trying such an approach to ensure compatibility; In images with no relation between objects, I am using a mock ground truth triplet of [box0, no_relation, box0], which is [[0,0,0]].

However, I wonder if that negatively impacts the learning process, since this way, in images without relations we explicitly ask and supervise the network to produce this "no_relation" relation. And when actual relations exist, we do not have this.

I'd appreciate if you have any insight on this. Thanks!