search

shubhamagarwal92 / mmd

This repository contains the Pytorch implementation for our SCAI (EMNLP-2018) submission "A Knowledge-Grounded Multimodal Search-Based Conversational Agent"
29 stars 5 forks source link

Guidance for changing image feature extractor #4

Closed hardik2396 closed 5 years ago

hardik2396 commented 5 years ago

I want to change image feature extractor from VGG-16 to Resnet 50 or something.

Thanks in advance.

shubhamagarwal92 commented 5 years ago

Hi Hardik,

In the pickle dataset that we use, each image is assigned a file path (hard coded by Shah et al.) and we use the annoy index to retrieve the image vectors (based on this file path) as done by Shah et al.

You would have to:

  1. Align the image path with their actual urls from raw meta data.
  2. Crawl the images from the web
  3. Run any image extractor (pytorch provides direct api for that) and save it in (like h5) files. Or you can pass the raw image pixel values and extend the image_encoder if you want to fine tune as well.
hardik2396 commented 5 years ago

Thank you very much for your guidance.