search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.03k stars 54 forks source link

Module 'exact_match' doesn't exist on the Hugging Face Hub either. #18

Closed xienan0326 closed 3 months ago

xienan0326 commented 3 months ago

install: pip install lmms-eval

run: accelerate launch --num_processes=8 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b" --tasks mme --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/

bug: Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.10/dist-packages/lmms_eval/main.py", line 24, in from lmms_eval import evaluator, utils File "/usr/local/lib/python3.10/dist-packages/lmms_eval/evaluator.py", line 17, in import lmms_eval.api.metrics File "/usr/local/lib/python3.10/dist-packages/lmms_eval/api/metrics.py", line 146, in exact_match = evaluate.load("exact_match") File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 748, in load evaluation_module = evaluation_module_factory( File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 681, in evaluation_module_factory raise FileNotFoundError( FileNotFoundError: Couldn't find a module script at /exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

kcz358 commented 3 months ago

Hi, may I asked have you installed llava from github also? I did not encounter any issue using the same command. You may want to check if flash-attn and llava are both installed.

xienan0326 commented 3 months ago

Hi, may I asked have you installed llava from github also? I did not encounter any issue using the same command. You may want to check if flash-attn and llava are both installed.

yeah，pip install package or pip install -e . Tried both installation methods flash-attn and llava are both installed. What else do I need to check? image

kcz358 commented 3 months ago

Hmm... , This is interesting. Mine works perfectly fine under the same environment.

Can you provide the whole running log of your program? @xienan0326

Maybe need to see where the error occurs. Is it before model responding or postprocessing.

lucasjinreal commented 3 months ago

Same error

/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

I would suggest try test your script on more robust envirment, this is hard to debug, it evolves logic inside.

xienan0326 commented 3 months ago

Same error

/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

I would suggest try test your script on more robust envirment, this is hard to debug, it evolves logic inside. 手动配置下可以，但是又遇到了如何加载本地数据的问题 https://huggingface.co/spaces/evaluate-metric/exact_match