Thanks for the repository and the paper (cool idea!).
I was wondering how I can reproduce your results with either GPT-3 or Flan-T5? What you show in the README is UnifiedQA, as far as I can see works without any few-shot demonstrations, and also from my experiments performs significantly worse on OK-VQA (around 32% even with a larger T5 than you use in the README).
Would I need to run https://github.com/Yushi-Hu/PromptCap/blob/main/new_pica/run_pica_okvqa.sh ?
If so, could you make the needed files available? They seem to be custom files that do not come with the standard datasets.
As in our README, we uploaded the experiment logs of GPT-3 in Evaluation Logs, containing all the prompts and GPT-3 answers. All results are generated by Codex. Hope it helps!
Hi!
Thanks for the repository and the paper (cool idea!).
I was wondering how I can reproduce your results with either GPT-3 or Flan-T5? What you show in the README is UnifiedQA, as far as I can see works without any few-shot demonstrations, and also from my experiments performs significantly worse on OK-VQA (around 32% even with a larger T5 than you use in the README). Would I need to run https://github.com/Yushi-Hu/PromptCap/blob/main/new_pica/run_pica_okvqa.sh ? If so, could you make the needed files available? They seem to be custom files that do not come with the standard datasets.
Thank you! Benno