search

Yushi-Hu / PromptCap

natual language guided image captioning
73 stars 7 forks source link

Reproduce paper results #4

Closed BennoKrojer closed 9 months ago

BennoKrojer commented 1 year ago

Hi!

Thanks for the repository and the paper (cool idea!).

I was wondering how I can reproduce your results with either GPT-3 or Flan-T5? What you show in the README is UnifiedQA, as far as I can see works without any few-shot demonstrations, and also from my experiments performs significantly worse on OK-VQA (around 32% even with a larger T5 than you use in the README). Would I need to run https://github.com/Yushi-Hu/PromptCap/blob/main/new_pica/run_pica_okvqa.sh ? If so, could you make the needed files available? They seem to be custom files that do not come with the standard datasets.

Thank you! Benno

Yushi-Hu commented 9 months ago

Sorry for the long wait!

As in our README, we uploaded the experiment logs of GPT-3 in Evaluation Logs, containing all the prompts and GPT-3 answers. All results are generated by Codex. Hope it helps!