search

facebookresearch / open-eqa

OpenEQA Embodied Question Answering in the Era of Foundation Models
MIT License
209 stars 20 forks source link

Baselines for open source models #2

Open kalyani7195 opened 5 months ago

kalyani7195 commented 5 months ago

Hello! Thank you for open-sourcing the code base. I was wondering if it is possible to release the code for open source models such as llama and llava. I unfortunately do not have access to gemini/ claude/ gpt APIs. So it would be great if I could run the open sourced baselines. Kalyani

arjunmajum commented 5 months ago

Quick update: we're adding code for LLaMA in PR #4.

asuglia-alana commented 5 months ago

Hello @arjunmajum,

On a related note, I believe that to aid reproducibility of your evaluation, it would be best to use an open-source model as an evaluator instead of using GPT-4 which could change and might be deprecated one day. Why not use something Llama-3 as an evaluator?

In general, I think that the best way to solve this issue is to release the predictions of the models you have used so that everybody can run the evaluation again with the latest and greatest model available. Would it be possible for you to release the predictions?

Thanks, Alessandro

asuglia-alana commented 5 months ago

Hello @arjunmajum any updates?