search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.33k stars 91 forks source link

[BUG] Potential bug in `process_docs()` function pipeline #194

Open wykang opened 1 month ago

wykang commented 1 month ago

The current pipeline has a potential bug when applying the process_docs() function.

The process_docs() function is invoked when the self.{split}_docs() function is called, as seen in the following lines:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/536b70b4e5797f8bc51d46bc658e05db1d2b3c3b/lmms_eval/api/task.py#L926-L930

However, within the fewshot_context() function, a doc is directly retrieved from self.dataset_no_image (line 952), which has not had process_docs() applied. As a result, the example retrieved in line 958 may not be in the intended format.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/536b70b4e5797f8bc51d46bc658e05db1d2b3c3b/lmms_eval/api/task.py#L952-L958

This could potentially lead to inconsistencies or errors in the document format during processing.

kcz358 commented 4 weeks ago

I believe in our current pipeline, you can not test few shot now. So maybe won't be an issue currently. But thank you for pointing that out, we will fix this if few shot testing is added to our repo