search

TinyLLaVA / TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models
https://arxiv.org/abs/2402.14289
Apache License 2.0
662 stars 69 forks source link

No module named 'tinyllava.eval.m4c_evaluator' #30

Closed soham-joshi closed 8 months ago

soham-joshi commented 8 months ago

Hey Team, thank you for the great work!

I was evaluating the model "bczhou/TinyLLaVA-3.1B" on Text-VQA. However, I am getting the following error. 

File "TinyLLaVABench/tinyllava/eval/eval_textvqa.py", line 6, in <module>
from tinyllava.eval.m4c_evaluator import TextVQAAccuracyEvaluator
ModuleNotFoundError: No module named 'tinyllava.eval.m4c_evaluator'

Seems like the class m4c_evaluator isn't present in the code. Could you please help in this issue? @baichuanzhou @jiajunlong @huangleiBuaa @eltociear

soham-joshi commented 8 months ago

Moreover, could you also help me with these files which is required for the evaluation?

--question-file $EVAL_DIR/textvqa/llava_textvqa_val_v051_ocr.jsonl \
--answers-file $EVAL_DIR/textvqa/answers/$MODEL_NAME.jsonl \

@baichuanzhou @jiajunlong @huangleiBuaa @eltociear

Thanks!

baichuanzhou commented 8 months ago

Check out our evaluation script for TextVQA here. Follow instructions here to download all relevant data.

soham-joshi commented 8 months ago

Okay, thank you for your response. I am using the same script for evaluation. I have already downloaded the eval.zip from the above link which you shared. The folder contains: textvqa/llava_textvqa_val_v051_ocr.jsonl textvqa/answers/llava-v1.5-13b.jsonl

Could you clarify what do the arguments represent in the above script which you shared?

  1. --question-file
  2. --answers-file
  3. --result-file
baichuanzhou commented 8 months ago

Sure.

  1. --question-file. We use textvqa/llava_textvqa_val_v051_ocr.jsonl as the question file. This file contains prompt questions and necessary meta information for evaluating TextVQA. Every element in that json file contains: question_id, image, text, category. You can check it out yourself.
  2. --answer-file. We prompt the model with questions from the question_file, and the result will be written to the --answer-file. As we have trained a lot of different models, we set different $MODEL_NAME to ensure their result files won't overlap, and the textvqa/answers/llava-v1.5-13b.jsonl is a baseline answer file of the LLaVA-1.5-13B model.(https://huggingface.co/liuhaotian/llava-v1.5-13b).
  3. --result-file. This is the args we pass into the tinyllava/eval/eval_textvqa.py. We compared the output generated by our model with ground truth answers, which is provided in TextVQA_0.5.1_val.json.

I hope I have answered your questions. If you find our repo useful, please consider giving us a star. Thanks.

soham-joshi commented 8 months ago

Thank you @baichuanzhou for your patience and the detailed response as well. Yes, I have got my query answered, thanks! Closing this issue.

soham-joshi commented 8 months ago

I am using the same m4c_evaluator.py as this https://github.com/haotian-liu/LLaVA/blob/main/llava/eval/m4c_evaluator.py.

cooleel commented 6 months ago

It's actually missing a file.