question about VG-Relation categories

[hiker-lw]

Hi, thank you for your wonderful work! I found that there are there VG-Relation categories (pulled by, pulling and leaning against) produced by your jupyter notebook (Replicate ARO! VG-Relation, VG-Attribution.ipynb) that not appeared in the paper.

[mertyg]

Thanks for your interest and kind words @hiker-lw !

I think we just did not exclude them in the notebook here, but there are relations with less than 25 samples that are included in the data frame. The ones you suggested are some of those relations.

@vinid explained it here, i'll quote:

We remove those relations for which we had too few to report something that could be significant: I think some attributes/relations occur a couple of times, however, NegCLIP or CLIP (or the other) could get this right just by chance and this can significantly shift the macro accuracy. Thus we decided to remove those under a certain threshold.

[hiker-lw]

Thanks for your kind reply! I understand what you said, excluding these sparse samples is a fair and more convicing procedure, but there are many relations (particularly the verbs relations) with less than 25 sampls, which are reported in the paper. So it's a little confusing me, looking forward for your reply ~

[mertyg]

Oh I'm sorry you're right -- the number is 10, not 25. If you check out the relations that appear in the released dataset and not in the paper, you'd see the following:

At any rate, I think it'll be good for us to clarify this in the paper/repo, thanks for the feedback!

[hiker-lw]

Thanks for your kind reply！I understand the situation now. Thank you so much for great work again! I will close this issue.