search

cshizhe / hgr_v2t

Code accompanying the paper "Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning".
MIT License
209 stars 21 forks source link

If there is no verb in the sentence, what should we do with it？ #25

Closed lDarryll closed 2 years ago

lDarryll commented 3 years ago

If there is no verb in the sentence, what should we do with it

cshizhe commented 3 years ago

The model combines cross-modal scores from three levels (the global sentence, verbs, and noun phrases). So when there is no verb in the sentence, the other two levels would be used for cross-modal matching.

lDarryll commented 3 years ago

image However, if there are no verbs, an error will be prompted when generating the graph. Do I ignore it or eliminate the sentence without verbs, or add verbs manually

cshizhe commented 3 years ago

Does your error come from the "assert" in the red rectangle? This assert line is to ensure that the number of tokenized words from spacy and that from srl toolkit are the same. Otherwise the word indexes are not aligned and we cannot get correct results. In this case, you probably should not augment the parsed graph automatically. Manual process should be fine.

lDarryll commented 3 years ago

Do you have any other augment methods or tools besides manual augment ? In addition, is it necessary to augment the graph? If not, can I use the non enhanced graph?

cshizhe commented 3 years ago

In my case, most of the sentences can be correctly parsed, so I didn't do any manual process and the performance was good. I think it depends on your data how to augment the graph. You may need other NLP parsing tools if there are too many failure cases.