search

OFA-Sys / OFA

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Apache License 2.0
2.39k stars 248 forks source link

Regarding fine tuning for Custom VQA dataset #408

Open manas6266 opened 1 year ago

manas6266 commented 1 year ago

Thanks for your great work, I am interested in fine-tuning OFA for Visual Question Answering (VQA) using my custom dataset, which includes image-question-answer pairs. However, my dataset lacks confidence scores for the answers. I would like to understand why confidence scores are needed for OFA fine-tuning and how I can handle this absence in my case. Additionally, I've noticed that even the VQA-v2 dataset does not include confidence scores. During inference, will the answers be generated from fixed vocabulary pickle files only, and if so, what is the reason for not using classification models instead of OFA?

logicwong commented 12 months ago

@manas6266

  1. You can set the confidence score of each answer to 1.
  2. In the original vqav2 dataset, each sample contains multiple answers. We followed the previous works to set the confidence score for each answer based on its frequency.
manas6266 commented 11 months ago

What is the max token size we could give to model?

On Thu, 31 Aug 2023 at 20:05, Wang Peng @.***> wrote:

@manas6266 https://github.com/manas6266

  1. You can set the confidence score of each answer to 1.
  2. In the original vqav2 dataset, each sample contains multiple answers. We followed the previous works to set the confidence score for each answer based on its frequency.

— Reply to this email directly, view it on GitHub https://github.com/OFA-Sys/OFA/issues/408#issuecomment-1701173057, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOWGQIFX5Q2WDEEZCD2MTF3XYCOMVANCNFSM6AAAAAA2YFOM2I . You are receiving this because you were mentioned.Message ID: @.***>