Closed kangzhao2 closed 2 years ago
Hi @kangzhao2 ,
The "sequence length" warning is related to the tokenizer, which we also observed and should not influence the results.
I'm not sure why is that, we use the same GPU/Pytorch setting. One random guess is the used OCR feature? As I remember a ~4% improvement by using MSOCR.
Meanwhile, please feel free to provide more guesses/observations related to this. Thanks
Dear authors:
I download the checkpoints Model checkpoints (~17G) and evaluate the model using the following code:
python tools/run.py --tasks vqa --datasets m4c_textvqa --model m4c_split --config configs/vqa/m4c_textvqa/tap_refine.yml --save_dir save/m4c_split_refine_test --run_type val --resume_file save/finetuned/textvqa_tap_base_best.ckpt
I got the following results:
And I found an error prompt during the evaluation:
In my opinion, the accuracy should be 0.4991 as shown in the following table:
What's wrong with my operations? Is there something to do with the error I encounter?
By the way, when I use the OCR-CC checkpoints: save/finetuned/textvqa_tap_ocrcc_best.ckpt, the accuracy is 0.4934 (which should be 0.5471), and I found the same error as mentioned above.
The GPU and PyTorch version is as following:
Hope to get your response
Thanks