search

gqa-ood / GQA-OOD

GQA-OOD is a new dataset and benchmark for the evaluation of VQA models in OOD (out of distribution) settings.
27 stars 1 forks source link

Maybe ood-testdev in LXMERT is more flexible #3

Closed alice-cool closed 2 years ago

alice-cool commented 2 years ago

Dear scholar, Because we always finetune the model in downstream tasks. But you select the validation ood split to define alpha. It is not flexible because it needs pretraining again by excluding the qa pairs from validation sets. Just suggestions.

gqa-ood commented 2 years ago

Dear @alice-cool,

Yes, I agree with you that it is not flexible to have to do the pretraining again. But, we choose to validate on the validation and test on the test-dev because the GQA test-std is not public, so we cannot create a gqa-ood test-std set.

At the same time, I do not really understand why qa pairs from the validation set are used for training!

Corentin K.

alice-cool commented 2 years ago

Dear scholar, It is just the default setting for lxmert with high performance. Maybe you think 12587 samples from testdev is not good to make gqa-ood testdev set. image

If you didn't train the validation set on training phase. so the LXMERT default hypermeter will be changed with validation accuracy, or wasn't changed just excluding validation set for training. Because I didn't find any hypermeter for LXMERT in gqa-ood testdev accuracy 49.8 at tail.

image