search

showlab / Awesome-MLLM-Hallucination

📖 A curated list of resources dedicated to hallucination of multimodal large language models (MLLM).
268 stars 8 forks source link

New paper for hallucination benchmarks #5

Closed LFhase closed 2 weeks ago

LFhase commented 2 weeks ago

Hi, thanks for curating and maintaining this amazing list of MLLM hallucination! We would like to contribute our work released a few months ago to this list:

We trace back the hallucination issue to the multimodal alignment in CLIPs, which are often incorporated as MLLM visual encoders. We find surprising phenomena that CLIPs demonstrate high vulnerability to the natural distribution shifts captured by CounterAnimal and perform even worse than previous ImageNet-based models. This issue further leads to severe hallucinations of MLLMs such as LlaVA and MiniGPT-4.

BTW, we also released a project page with the link to our benchmark, and feel free to decide what is the best way to add/or not add it to the list : ) Page

Thank you very much!

JosephPai commented 2 weeks ago

Hi @LFhase , thanks for your interests in this repo. I recently refactored this paper list a bit. Your paper has been added to the section of Hallucination Evaluation & Analysis. Very interest work! Hope it helps.