search

linjieli222 / VQA_ReGAT

Research Code for ICCV 2019 paper "Relation-aware Graph Attention Network for Visual Question Answering"
https://arxiv.org/abs/1903.12314
MIT License
178 stars 38 forks source link

Some question about semantic relationship classification #26

Closed HeartFu closed 3 years ago

HeartFu commented 3 years ago

Hi author,

I find your paper is very interesting and want to use it on my model. But I have some questions about semantic relationship classification.

You said you have top 15 relationships after normalizing the predicates with relationship-alias, like wearing, holding, can I ask:

  1. How do you do with the other relationships which are not included in Top 15 sets? Set them as the non-relation or drop out them? If you set them to the non-relation labels, the number of the non-relation sets could be far more than other labels. How do you deal with this problem?

Thanks.

linjieli222 commented 3 years ago

Hi,

Thanks for your interests in this project and sorry about the late reply.

We include an additional non-relation class when classifying the relations between two objects.

It is true that the non-relation can be far more than other labels. However, when choosing the top 15 relationships, it covers more than 80% of the instances in the Visual Genome (VG). As you may find in visual relation detection or scene graph generation papers, the distribution of relations in VG is very long-tailed.

In addition, even if the predicted relations is non-relation, we still perform GAT on top of it, only the label embeddings will be different.

Thanks, Linjie