Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering (ACL21)

[hideaki-j]

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

Contribution summary

• Kim et al. proposed the CQA method which jointly encodes the raw and rewritten question
• in order to infuse explicit guidance on resolving conversational dependencies,
• and achieved +1.2 F1 on both QuAC and CANARD.

Authors

Gangwoo Kim, Hyunjae Kim, Jungsoo Park, and Jaewoo Kang (Korea University)

Motivation

Two types of existing methods: (1) end-to-end and (2) QR-based pipelined model.

• In (1), the end-to-end model jointly encodes the evidence document, the current question, and the whole conversation history. However, the model has limitations to do so without explicit guidance on how to resolve these dependencies.
• In (2), QA models are dependent on QR models; hence QA models suffer from the error propagation from QR models

Method

• Proposed "unified framework": method jointly encode them to train QA models with consistency regularization.
• Specifically, when original questions are given, QA models are encouraged to yield similar answers to those when self-contained questions are given

Results / Insight

• Main results:
  • Improves the QA performance +1.2 F1 on QuAC (RoBERTa, end-to-end) and CANARD (RoBERTa, pipelined)
  • QR did not work well on QuAC, where human rewritten query is not available for train QR
  • What hideaki-j learned: if human rewritten query is not available, then go for end-to-end approach.
• Transferability:
  • Tested on CoQA, where question distributions are significantly different from QuAC and CANARD.
  • Results: improved QA performance (+0.6 for RoBERTa, CoQA)

[hideaki-j]

Question

In introduction, they claimd that

EXCORD significantly improves the QA models’ performance by up to ... 5.2 F1 on CANARD.

However, when compared to the "pipeline" (i.e., QR+QA) model, which is also should be the baseline model, then the performance improvement is only +0.4.

[hideaki-j]

Related reference

• AMBIGQA: Answering Ambiguous Open-domain Questions
  • clarifying ambiguous questions using QR
  • In related work: "In recent work, Xu et al. (2019) study clarification of questions that are intentionally annotated with pre-specified entity reference ambiguities."