Text entailment trees

[katesanders9]

Overview

Goal: Write the program that generates entailment trees for TVQA using dialogue only, and write corresponding evaluation scripts to assess performance.

Progress

Filters

• [x] Add entailment evaluation script
• [x] Identify relevant datasets for filters
• [x] Implement SQuAD-based RTE filter
• [ ] Evaluate SQuAD filter and identify failure modes
• [ ] Add additional RTE filters based on SQuAD filter performance

  Search

• [x] Implement base search algorithm
• [ ] Implement entailment tree evaluation
• [ ] Evaluate full text pipeline

Evaluation

TBD

[katesanders9]

Filter selection notes

Model architectures

Presently, the architectures are kept to SBERT (for cosine similarity analysis) and CrossEncoder classifiers, both from the SentenceTransformers package.

Datasets

Entailment datasets

• EntailmentBank
• SNLI
• MultiNLI: SNLI for a variety of genres, including dialogue (without name labels)
• WatClaimCheck: Claims from a variety of genres
• QNLI: SQuAD subset transformed into QA-based NLI
• DialogueNLI: NLI for dialogue only, mostly "I" statements

Note: Here is a paper on transforming QA datasets into NLI datasets.

QA datasets

Ideally, the datasets used to train the filters will follow the general paradigm of (question, evidence dialogue) pairs, corresponding to a larger document of dialogue exchanges. However, there are a limited number of datasets that fall into this category. Relevant QA datasets are listed below:

• SQuAD (1.0/2.0): Question - text segment pairs mapped to Wikipedia text documents.
• QuAC and CoCA: Question - text segment pairs corresponding to a text document, but the questions and answers are in dialogue form.

Note: The domain shift between SQuAD, QuAC, and CoCA is notable, and so code to convert data between the three formats has been published.

• NarrativeQA: Question - text segment pairs for movie scripts + stories.
• CBT: QA pairs with evidence from children's books.
• DuoRC: Question - answer pairs with text segments from movie summary documents.
• Google Natural Questions: Question - text segments/paragraphs mapped to Google search results.
• RACE: Reading comprehension questions for dialogue-heavy narrative documents.
• DROP: Question - multispan answers from text documents. Questions require different types of reasoning using the multiple spans.
• HotPotQA: Multiple supporting facts per question

Note: Here's another repository that converts data between some more of the above datasets.

Dialogue datasets

Other datasets exist that are dialogue-centric, but instead of including specific QA pairs that can be answered by a specific line of dialogue, the dialogue lines themselves are annotated for various attributes. These could feasibly be preprocessed using T5/etc. to turn them into dialogue-centric QA datasets.

• DailyDialogue
• FriendsCorpus