search

matt-seb-ho / WikiWhy

WikiWhy is a new benchmark for evaluating LLMs' ability to explain between cause-effect relationships. It is a QA dataset containing 9000+ "why" question-answer-rationale triplets.
MIT License
44 stars 1 forks source link
artificial-intelligence dataset explainable-ai iclr2023 machine-learning nlp nlp-datasets open-domain-qa question-answering

readme

WikiWhy

Paper | Poster

WikiWhy is a new benchmark for evaluating models' ability to explain between cause and effect. WikiWhy is a QA dataset built around the novel auxiliary task of explaining the answer to a "why" questions in natural language. It contains over 9,000 “why” question-answer-rationale triples, grounded on Wikipedia facts across a diverse set of topics. Each rationale is a set of supporting statements connecting the question to the answer.

This paper was accepted as a top 5% paper with oral presentation to the International Conference on Learning Representations (ICLR 2023) in Kigali, Rwanda.

Figure 1.

Dataset Usage

In light of data contamination concerns, and to prevent WikiWhy from inadvertently being included in pre-training corpora, we've separated WikiWhy's columns into separate files. We hope that separating the inputs and output labels can help preserve WikiWhy's value as a benchmark. To load WikiWhy into a single dataframe, we provide a simple script in /code/load_wikiwhy.py. Either copy the function source or import into your code.

from load_wikiwhy import load_wikiwhy
wikiwhy = load_wikiwhy(directory_path="../dataset/v1.1/")

Updates

Citation

@inproceedings{
    ho2023wikiwhy,
    title={WikiWhy: Answering and Explaining Cause-and-Effect Questions},
    author={Matthew Ho and Aditya Sharma and Justin Chang and Michael Saxon and Sharon Levy and Yujie Lu and William Yang Wang},
    booktitle={International Conference on Learning Representations},
    year={2023},
    url={https://openreview.net/forum?id=vaxnu-Utr4l}
}