search

facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.49k stars 935 forks source link

TextVQA accuracy on val set is different for different batch size #1253

Closed assjdshisio closed 2 years ago

assjdshisio commented 2 years ago

❓ Questions and Help

assjdshisio commented 2 years ago

When I run the M4C code, the val accuracy is different if the batch size could be not divisible by the size of val set. For instance, There are 5000 questions on the TextVQA val set, if the batch size is 100, the result is always same. If the batch size could not be divisible by 5000 (e.g. 128 or 64), the results are different. The final result is obtained by taking an average of accuracies of all batches. However, the last bacth size is smaller than others, which may lead to the error.

assjdshisio commented 2 years ago