search

TinyLLaVA / TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models
https://arxiv.org/abs/2402.14289
Apache License 2.0
658 stars 68 forks source link

Is it convenient to provide some JSON files #18

Closed wangfengjuan closed 8 months ago

wangfengjuan commented 8 months ago

image Good job, thank you for sharing, but during the verification process, I need some files. Would it be convenient for you to provide them? val_predictions.json and val_attentions.json thankyou！

baichuanzhou commented 8 months ago

Which eval process is this?

wangfengjuan commented 8 months ago

Which eval process is this? image

When performing gqa validation, these two files are missing in the eval. py file

baichuanzhou commented 8 months ago

Please provide the full error message so I could have an idea of what is really missing here.🤓

wangfengjuan commented 8 months ago

Please provide the full error message so I could have an idea of what is really missing here.🤓

image Missing this file

baichuanzhou commented 8 months ago

It’s a little confusing since I don’t know what exactly happened. Did you following the instructions in Evaluation.md to download eval data first? What script were you running? Eval scripts should be self-contained so I need to have a clearer idea of what you did.

wangfengjuan commented 8 months ago

It’s a little confusing since I don’t know what exactly happened. Did you following the instructions in Evaluation.md to download eval data first? What script were you running? Eval scripts should be self-contained so I need to have a clearer idea of what you did.

I have downloaded the relevant data required for validation on the GQA dataset and now run the script: CUDA_VISIBLE-DEVICES=0 bash scripts/tiny_llava/eval/gqa. sh. Then there was a problem of missing the following files, which were not included in the downloaded validation dataset or the data you provided.Looking forward to your reply, thank you!

I have downloaded the relevant data required for validation on the GQA dataset and now run the script: CUDA_VISIBLE-DEVICES=0 bash scripts/tiny_llava/eval/gqa. sh. Then there was a problem of missing the following files, which were not included in the downloaded validation dataset or the data you provided. Looking forward to your reply, thank you! image

baichuanzhou commented 8 months ago

The error message you provided indicates that you ran python eval/eval.py --tier val instead of python eval/eval.py --tier testdev_balanced in your gqa.sh file. Please change back to python eval/eval.py --tier testdev_balanced so that evals can be run correctly. If you want to run evaluation on the validation split of gqa, the eval.zip should provide you with val_all_questions.json and val_balanced_questions.json in your gqa eval directory, use them to run eval.

Let me know if you have further questions.

baichuanzhou commented 8 months ago

This issue will be closed in 1 day due to inactivity.