search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.
https://lmms-lab.framer.ai/
Other
2.11k stars 165 forks source link

DocVQA Eval: IndexError: Invalid key: 5205 is out of bounds for size 5188 #193

Open SachinG007 opened 3 months ago

SachinG007 commented 3 months ago

Facing the error below while evaluating on the docVQA dataset.

Traceback (most recent call last):
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/__main__.py", line 206, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/__main__.py", line 301, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/utils.py", line 453, in _wrapper
    return fn(*args, **kwargs)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/evaluator.py", line 135, in simple_evaluate
    results = evaluate(
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/utils.py", line 453, in _wrapper
    return fn(*args, **kwargs)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/evaluator.py", line 297, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/models/llava.py", line 303, in generate_until
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/models/llava.py", line 303, in <listcomp>
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2800, in __getitem__
    return self._getitem(key)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2784, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 5205 is out of bounds for size 5188
hxhcreate commented 1 month ago

+1, while using qwen2-vl

pengzhansun commented 4 days ago

Hi, have you figured out what is the bug?

chadlzx commented 4 days ago

I have also meet this problem when I test qwen2-vl-7B-instruct on textVQA with batch_size != 1.

chadlzx commented 3 days ago

oh, I find this issue. I think this info should be highlight.