search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.03k stars 53 forks source link

03-25 13:08:56 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating #30

Open awzhgw opened 3 months ago

awzhgw commented 3 months ago

03-25 13:09:00 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating 03-25 13:09:03 [lmms_eval/models/llava.py:279] ERROR Error Invalid key: 3139 is out of bounds for size 2801 in getting visuals, task_type is multidocvqa_test) Traceback (most recent call last): File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 199, in cli_evaluate results, samples = cli_evaluate_single(args) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 283, in cli_evaluate_single results = evaluator.simple_evaluate( File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper return fn(*args, kwargs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate results = evaluate( File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper return fn(*args, *kwargs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate resps = getattr(lm, reqtype)(cloned_reqs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until raise e File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until visuals = [doc_to_visual0 for ids in doc_id] File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in visuals = [doc_to_visual0 for ids in doc_id] File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in getitem return self._getitem(key) File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem pa_subtable = query_table(self._data, key, indices=self._indices) File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table _check_valid_index_key(key, size) File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key raise IndexError(f"Invalid key: {key} is out of bounds for size {size}") IndexError: Invalid key: 3139 is out of bounds for size 2801 03-25 13:09:03 [lmms-eval/lmms_eval/main.py:213] ERROR Error during evaluation: Invalid key: 3139 is out of bounds for size 2801 Traceback (most recent call last): File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 199, in cli_evaluate results, samples = cli_evaluate_single(args) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 283, in cli_evaluate_single results = evaluator.simple_evaluate( File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper return fn(args, kwargs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate results = evaluate( File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper return fn(*args, **kwargs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate resps = getattr(lm, reqtype)(cloned_reqs) File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until raise e File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until visuals = [doc_to_visual0 for ids in doc_id] File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in visuals = [doc_to_visual0 for ids in doc_id] File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in getitem return self._getitem(key) File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem pa_subtable = query_table(self._data, key, indices=self._indices) File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table _check_valid_index_key(key, size) File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key raise IndexError(f"Invalid key: {key} is out of bounds for size {size}") IndexError: Invalid key: 3139 is out of bounds for size 2801

@ChunyuanLI @Luodian @FengLi-ust @jzhang38 @JvThunder how to resolve it ?

Luodian commented 3 months ago

Please stick the transformers version to 4.37.2

We would have later PR to make it compatiable with transformers > 4.37.2