Precomputed visual features pytorch FasterRcnn VG

[AndresPMD]

Hello @kuanghuei ,

First congrats for your paper and repo. Since your repo uses the precomputed visual features from the Bottom-up model, but sadly if somebody wants to use your model in another dataset they have to install caffe (sometimes painful). I have found a repo: https://github.com/shilrley6/Faster-R-CNN-with-model-pretrained-on-Visual-Genome

That offers the Faster RCNN trained on VG as well as the data preprocessing toolkit (tsv and numpy) to extract features on another dataset on pytorch :) .

Maybe this info is helpful for any other that runs into the same problem. You can either add this info in your Readme as an optional way of extracting the visual features.

I am testing it out myself atm.

Cheers.

[LetsGoFir]

Hi, does it work?

[AndresPMD]

It is the same model as the one the repo uses to extract but rather than dealing with caffe (can be painful) it is on pytorch. Check that repo and read the code :)

[LetsGoFir]

It is the same model as the one the repo uses to extract but rather than dealing with caffe (can be painful) it is on pytorch. Check that repo and read the code :)

I have run it by myself, and the feature is different with the author's. In fact, it cannot converge when training. Have you try it?

[AndresPMD]

@LetsGoFir Yes I have tried. The training seems to converge but the results are not as good as the ones with the original caffe feats. Performance on Flickr30K (Paper) (R@1, R@5, R10, Caption Retrieval and Image Retrieval) VSRN (ours)71.3 90.6 96.0 54.7 81.8 88.2

What I got:

63.6 | 86.4 | 92.1 47.6 | 75 | 83.3

If you find some other good repo to use as feature extractor let me know.

Good luck!

Probably there is another repo that implements this Faster R CNN on VG in a better manner.

[LetsGoFir]

@LetsGoFir Yes I have tried. The training seems to converge but the results are not as good as the ones with the original caffe feats. Performance on Flickr30K (Paper) (R@1, R@5, R10, Caption Retrieval and Image Retrieval) VSRN (ours)71.3 90.6 96.0 54.7 81.8 88.2

What I got:

63.6 | 86.4 | 92.1 47.6 | 75 | 83.3

If you find some other good repo to use as feature extractor let me know.

Good luck!

Probably there is another repo that implements this Faster R CNN on VG in a better manner.

Sign, my training cannot converge, can you give me your f30k_test.json and f30k_test.npy? I want to know where it goes wrong. And I am extracting the feature by bottom-up-attention caffe, but with python3. The feature seems like different with the author's, again. But the shapes match well. :(

[AndresPMD]

Well atm I don't have the npy file with the features. Nonetheless, in literature the most common split used in Flickr30K is the Karpathy split (https://cs.stanford.edu/people/karpathy/deepimagesent/). You can donwload the anns files from the link,

Good luck

[LetsGoFir]

Well atm I don't have the npy file with the features. Nonetheless, in literature the most common split used in Flickr30K is the Karpathy split (https://cs.stanford.edu/people/karpathy/deepimagesent/). You can donwload the anns files from the link,

Good luck

I have tried Anderson's caffe feature and it is good.

[184446223]

你好，有效果吗？

hello , Do you have the image bboxs of the features of Flickr30k

[184446223]

好吧，atm 我没有具有这些功能的 npy 文件。尽管如此，在文献中，Flickr30K 中最常用的拆分是 Karpathy 拆分 ( https://cs.stanford.edu/people/karpathy/deepimagesent/ )。 您可以从链接下载 anns 文件， 祝你好运

我已经尝试过 Anderson 的咖啡功能，它很好。

I hope the bbox is be set in the faster-rcnn Caffe