search

Cadene / murel.bootstrap.pytorch

MUREL (CVPR 2019), a multimodal relational reasoning module for VQA
https://arxiv.org/abs/1902.09487
BSD 3-Clause "New" or "Revised" License
194 stars 24 forks source link

Evaluation on test set #9

Closed archanray closed 5 years ago

archanray commented 5 years ago

Hi Cadene,

I used the following command to generate a JSON for the test set. python -m bootstrap.run \ -o logs/vqa2/murel/options.yaml \ --exp.resume best_accuracy_top1 \ --dataset.train_split \ --dataset.eval_split test \ --misc.logs_name test The JSON was generated in folder logs/vqa2/murel/results/test/epoch,1/. But when I upload this in EvalAI http://evalai.cloudcv.org/web/challenges/challenge-page/163/submission, it says failed (I did upload OpenEnded_mscoco_test-dev2015_model_results.json)! Please let me know if I am doing something wrong.

I have attached the log file (I couldn't attach the JSON as github doesn't support sharing JSON files, update: here is the link to the JSON file I uploaded in EvalAI: https://evalai.s3.amazonaws.com/media/submission_files/submission_25691/ab81c6df-954e-4436-80d2-12826b8ffa0c.json). Thanks for the help!

logs_test.txt

Cadene commented 5 years ago

@lousylory thanks for your interest

You should upload the test json not the test-dev json.

The test-dev json was useful in the first versions of the VQA challenge.

archanray commented 5 years ago

That worked! Thanks!

AmingWu commented 5 years ago

@LousyLory ， I want to reproduce the result of VQA-CP, which dataset should I download?