Spot_Difference
|-- bottom-up-attention-vqa
|-- checkpoints
|-- pretrained
|-- bert-base-uncased
|-- gpt2
|-- model_LXRT.pth
|-- ...
|-- data
|-- 0206
|-- spot_diff_train.json
|-- ...
|-- img_feat_3ee94.h5
|-- dataloader
|-- guesser_dataloader.py
|-- loader_utils.py
|-- qgen_dataloader.py
|-- lxmert
|-- ...
|-- model
|-- guesser.py
|-- qgen.py
|-- scripts
|-- stat_tools
|-- ...
Setup the environment by running pip install -r requirements.txt
.
The pre-trained model should be put in checkpoints/pretrained.
require to modify
GPT and LXMERT-based VQG model
sh scripts/train_<vqg_model_type>_vqg.sh
BUTD and LXMERT-based VQA model
sh scripts/train_<vqa_model_type>_vqa.sh
sh scripts/train_guesser.sh
sh scripts/self_play_{vqg_model_type}_{vqa_model_type}.sh