Create user stories the library / visualization

[anton164]

cc @dleve123 @jxmorris12 @cephcyn

• What will the library be used for?
• What will the visualization be used for?
• Who is the target audience?

[anton164]

First stab at a user story-ish:

User wants to explore whether a model generates summaries which are factual/faithful to the source documents

Given a set of generated summaries:

• Enable user to compute & inspect summarization metrics
• • Especially those which have been used for evaluting factuality/faithfulness (i.e. textual entaillment / question answering)
• Enable user to explore the summaries with metrics & annotations (if available)
• Enable user to do deeper analysis of annotations
• • Why are generated summaries not factual / unfaithful?
• • Are there classes of errors? (support deeper annotations)

Live inference

• Enable user to specify a model+document & generate a summary
• • Visualize inferred textual entailment task
• • Visualize question answering task

[dleve123]

Thinking at a slightly lower level of abstraction:

xsum example == xsum summary and document

1. User should be able to view xsum examples by filtering by id (done)
2. User should be able to view matching xsum examples by querying against summaries (in progress)
3. User should be able to view factuality/faithfulness metadata of xsum examples in context (in review)
4. User should be able to generate new summaries using summarization models (in-progress (model), todo (ui))
5. User should be able to generate summaries for arbitrary text using summarization models (todo)
6. User should be able to view summary quality via textual entailment metrics (in progress?)
7. User should be able to view summary quality via question answering tasks (in progress?)

[jxmorris12]

Thanks @anton164 👍