search

microsoft / TAP

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)
MIT License
70 stars 11 forks source link

TextVQA accuracy #29

Open crx15215682925 opened 8 months ago

crx15215682925 commented 8 months ago

Hello, I have got the json file of TextVQA, can you tell me how to get the accuracy? Only the eval code for Textcaps is currently provided.

soonchangAI commented 8 months ago

For TextVQA validation accuracy:

2 GPU:

python -m torch.distributed.launch --nproc_per_node 2 TAP/tools/run.py --tasks vqa --datasets m4c_textvqa --model m4c_split \ --config $config \ --run_type val \ --save_dir $save_directory \ --resume_file $model.ckpt \ training_parameters.distributed True single GPU:

python TAP/tools/run.py --tasks vqa --datasets m4c_textvqa --model m4c_split \ --config $config \ --run_type val \ --save_dir $save_directory \ --resume_file $model.ckp For TextVQA test accuracy, the test set label is not publicly release, So:

  1. Predict and save as JSON

python TAP/tools/run.py --tasks vqa --datasets m4c_stvqa --model m4c_split \ --config $config \ --run_type inference --evalai_inference 1 \ --save_dir $model.ckpt

  1. Upload to https://eval.ai/web/challenges/challenge-page/874/ for accuracy calculation
seuppp commented 7 months ago

Hi @soonchangAI, thanks for your reply.

Now I find the address provided in data/README.md is invalid. Could you please provide me with a new dataset download address? Thanks.

seuppp commented 7 months ago

Hi @crx15215682925

I find the address provided in data/README.md is invalid. Could you please provide me with a new dataset download address? Thanks.