search

TinyLLaVA / TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models
https://arxiv.org/abs/2402.14289
Apache License 2.0
662 stars 69 forks source link

Data about evaluation #75

Closed Fantasy1120 closed 5 months ago

Fantasy1120 commented 5 months ago

Hi, thanks for your excellent work. As I am downloading the evaluation data, I have two questions.

  1. The "testdev_balanced_predictions.json" in the GQA dataset looks like it should be the predictions?
  2. What is the coco dataset in the MME? Since you only mention "Downloaded images to MME_Benchmark_release_version", I'm not sure if I need to download the coco dataset additionally? If so, what version of the coco dataset and can you provide a link to it? Thanks again. eval_set
jiajunlong commented 5 months ago

Thank you for your reminder. Regarding the first issue, the "testdev_balanced_predictions.json" file is the output file of the model and does not exist before evaluation. For the second issue, there is no COCO dataset under MME. Both issues were due to errors in our documentation, and we have now completed the corrections.

Fantasy1120 commented 5 months ago

Thank you for your response.