DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Created by Hyeonwoo Noh, Paul Hongsuck Seo and Bohyung Han at POSTECH cvlab

Project page: [http://cvlab.postech.ac.kr/research/dppnet/]

Introduction

DPPnet is state-of-the-art Image Question Answering algorithm using dynamic parameter prediction to handle various types of questions.

Detailed description of the system will be provided by our technical report arXiv tech report

Citation

If you're using this code in a publication, please cite our papers.

@article{noh2015image,
  title={Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction},
  author={Noh, Hyeonwoo and Seo, Paul Hongsuck and Han, Bohyung},
  journal={arXiv preprint arXiv:1511.05756},
  year={2015}
}

Licence

This software is for research purpose only. Check LICENSE file for details.

System Requirements

• This software is tested on Ubuntu 14.04 LTS (64bit).
• At least 12GB gpu memory is required (NVIDIA tital-x gpu is used for training).

Dependencies

• torch [https://github.com/torch/distro]
• loadcaffe [https://github.com/szagoruyko/loadcaffe]
• xxhash [install: luarocks install xxhash]

Setup

Run "setup.sh" for setting up.

Testing

Scripts for testing is in "006_test_DPPnet". Use following commands for testing.

0. Run ./gen_simulinks.sh
1. Run th vqa_test.lua
2. Results will be saved in "006_test_DPPnet/save_result_vqa_test/results/"

Training

Following steps are required for training.

0. Train DPPnet with fixed cnn feature (004_train_DPPnet_fixed_cnn)
1. Finetune CNN from the model trained in the previous step (005_train_DPPnet_finetune_cnn)

Directories

• 001_porting_VQA_data: porting VQA data for torch implementation
• 002_extract_image_features: extracting VGG16 features from MSCOCO
• 003_skipthoughts_porting: porting model parameters for skipthoughts
• 004_train_DPPnet_fixed_cnn: training DPPnet with extracted feature
• 005_train_DPPnet_finetune_cnn: fine-tuning cnn from model trained 004
• cache: We cache loaded vqa questions ans vocabularies to reduce the time for loading
• data: data used for training / testing
• model: trained model parameters, model definitions, layer implementations
• utils: utilities (loading training data, loading models ...)