search

RUCAIBox / HaluEval

This is the repository of HaluEval, a large-scale hallucination evaluation benchmark for Large Language Models.
MIT License
377 stars 22 forks source link

Why are responses that ChatGPT refuses to answer considered as hallucinations? #21

Open JacksonWuxs opened 6 months ago

JacksonWuxs commented 6 months ago

Dear authors,

I found that in some cases where ChatGPT refused to answer, you were labeled as hallucinated. The following is a case from the General QA subset. {"ID": "815", "user_query": "Describe the painting using 3 adjectives.\n[painting.jpg]", "chatgpt_response": "As an AI language model, I do not have access to the picture of your choice, please provide me with a description of the painting so that I can offer 3 adjectives that match your description.", "hallucination": "yes", "hallucination_spans": ["As an AI language model, I do not have access to the picture of your choice, please provide me with a description of the painting so that I can offer 3 adjectives that match your description."]}

Would you please describe why these cases are considered as hallucination?

Thanks.

turboLJY commented 5 months ago

We are sorry for that. General QA dataset is labeled by humans, so there might be few little mistakes which are not checked by us. You can delete these cases from the datasets.