Probable bug in image retrieval code

[mods333]

Hi, I was trying to use the code for the image retrieval task. The code in 'dataloader.pyand 'preprocessing.py seem to be somewhat inconsistent. In dataloader.py there is _generate_tensor_by_idx that looks for 'image_graph and 'text_graph key in the img_txt_sg variable

However in the preprocessing code the output dictionary created had only two keys img and txt

I was wondering if there is something missing in the code or if I am doing something wrong.

[KaihuaTang]

sorry, my fault. I think 'image_graph' and 'test_graph' keys have been discarded, which means the following 4 lines can be removed from dataloader.py

[mods333]

@KaihuaTang I think those lines are required as in the scene graph encoding model (modelv2.py), the graph is used as attention weights.

From the code and the paper, what I understand is that the image_graph corresponds to the following equation in the paper.

I can add some code to compute the image_graph during preprocessing but I am not sure how the text_graph is to be generated

[KaihuaTang]

text_graph should be implemented the same as image_gaph. I didn't differentiate them in structure except for the vocabulary. sorry that I lost this part of codes. preprocessing.py was merged from several independent jupyter notes. I think I forgot to paste some functions.

[mods333]

I will try to figure out the missing part. Thanks for the help. Also thanks a lot for sharing this repository. Amazing work!

[cws7777]

@mods333 Have you figured out the missing part? because I'm kinda lost.. Thanks! :)