Pythia model usage with VizWiz

[1144181135]

I want to convert my dataset to the 'vizwiz' format and use Pythia model. I found several difficulties.

1. I have extracted features as the intuition, and get a folder composed of "xxx.npy" and "xxx_info.npy". How to use "tools/scripts/features/lmdb_conversion.py" to convert npy feature to lmdb feature. What is the argument "--mode"? What enviroment it dpends?

2. I use the original vizwiz dataset, download every thing automatically, and run "mmf_run config=projects/pythia/configs/vizwiz/defaults.yaml run_type=train dataset=vizwiz model=pythia". Assert Error happens: Traceback (most recent call last): File "/home/wulonglyx/miniconda3/envs/mmf/bin/mmf_run", line 33, in sys.exit(load_entry_point('mmf', 'console_scripts', 'mmf_run')()) File "/home/wulonglyx/competition/imagecaption/mmf/mmf_cli/run.py", line 133, in run main(configuration, predict=predict) File "/home/wulonglyx/competition/imagecaption/mmf/mmf_cli/run.py", line 56, in main trainer.train() File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/mmf_trainer.py", line 141, in train self.training_loop() File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 33, in training_loop self.run_training_epoch() File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 91, in run_training_epoch report = self.run_training_batch(batch, num_batches_for_this_update) File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 165, in run_training_batch report = self._forward(batch) File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 182, in _forward model_output = self.model(prepared_batch) File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/base_model.py", line 236, in call model_output = super().call(sample_list, *args, *kwargs) File "/home/wulonglyx/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, **kwargs) File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/pythia.py", line 294, in forward "image", sample_list, text_embedding_total File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/pythia.py", line 241, in process_feature_embedding "to number of features, {}.".format(len(feature_encoders), len(features)) AssertionError: Number of feature encoders, 2 are not equal to number of features, 1.

[apsdehal]

Hi, You can also directly use the npy features with MMF, no need to convert to LMDB. --mode=convert is what you are looking for when converting npy files to LMDB.

AssertionError: Number of feature encoders, 2 are not equal to number of features, 1.

Pythia model also requires ResNet152 grid features for which there is a script as well in tools. Alternatively, kick out the encoder from the model's config.

[apsdehal]

Closing this as resolved. If you have more questions, please open up a new issue.

[CCYChongyanChen]

@apsdehal I encountered a similar issue.

Overview

I can test the pretrained vqa2 model on vqa2 dataset; can train pythia model for the vizwiz dataset, but cannot test the pretrained vqa2 model on the vizwiz dataset.

command line I run

mmf_predict dataset=vizwiz model=pythia config=projects/pythia/configs/vizwiz/defaults.yaml run_type=test checkpoint.resume_file=/data/vqa/mmf/data/models/pythia_train_val.pth

The pythia_train_val.pth is provided by https://github.com/facebookresearch/mmf/issues/697#issuecomment-733942630

error message

The first error I encounter is

To address it, I referred to issue https://github.com/facebookresearch/mmf/issues/207 and changed mmf/mmf/configs/datasets/vizwiz/defaults.yaml' s answer_processor vocab_file from "vizwiz/v2019/extras/vocabs/answers_vizwiz_7k.txt" to "vqa2/defaults/extras/vocabs/answers_vqa.txt"

Then I encounter error

To address this, I referred this issue and removed "

• type: default params: model_data_dir: ${model_config.pythia.model_data_dir}" in pythia config.

Then I encounter

Thanks for the help in advance! You have resolved many issues posted by me hahaha, although I know many of the issues I posted are naive and silly. Thank you again. @apsdehal