We've had a request to get a bigger explanation of what we mean by "singleton" annotation and some reasons why we'd want it, so here's a case for singletons. Please post if you have disagreements/counter-proposals.
What we mean when we talk about singletons (let's focus on entities for now)
Singleton annotation means having a definition of something like "entity", and getting every entity regardless of whether it's repeated 2+ times in the document.
Loosely might link this to an idea like Kartunnen's (1976) "discourse referent" idea: "all the individuals, that is, events, objects, etc., mentioned in the text".
Generally all the "participants" in the document, plus other things passing some definition of "referentiality".
This can be contrasted to "Ontonotes" style: an Entity is something that is mentioned 2+ times in a document.
Even prominent participants (e.g. named entities) aren't Entities unless mentioned multiple times.
Loosely speaking: are we just doing linguistic coreference between mentions, or are we getting at some larger idea of what's "mentionable" or "an entity"?
Advantages of singleton annotation (system perspective)
The General "Anaphoricity" reason
A vague concensus of coreference people that you should do singleton when possible.
Based on some slides that I believe were in the Rethinking (Annotation of) Anaphora and Information Structure meetings:
Coreference systems still keep reporting that if they had gold mention detection, they'd perform much much better.
(That is complicated: the advantages of annotating singletons is slightly different from what those systems need. However, lots of ongoing coreference work is still focused issues that relate to these issues -- detecting "anaphoricity", "non-anaphoricity", singleton status, etc. (Wiseman et al '15, Recasens et al '13, etc.))
In short: Labeling singletons might help coreference systems.
Adapting Ontonotes-style mention detection systems to AMR might be a hard task; might provide clean targets for that task.
Advantages of singleton annotation (for actual representation purposes)
The general argument
Lots of "singleton" entities that you end up caring about on a document level
Entities with bridging relations (set/member, part/whole, etc) to a coreference chain
NEs with :wiki links but no other coreference links
Mentions with many mentions that all occur in a single sentence.
The KB / Cross-document question
Similarly, if you want to do cross-document annotation and already have all discourse referents, your task is to simply compare your entity chains and singletons (relatively small list)
Without that: any AMR concept could have a cross-document coreference link; must look at all AMRs again.
We can talk about "KB" tasks where the question is "who are the people and things discussed in this document"
If you want a set of things to feed into a system doing TAC / ERE / OpenIE, etc., what you want might be "a list of referents".
The "Annotation Quality" reason
There's a slight difference in approach to the actual annotation task when you're doing singleton vs non-singleton.
When doing singleton annotation, annotators tend to have at least one simple pass through a document labeling singletons.
This is the process you want anyways, if you want to catch short, hard-to-notice coreference chains
.- Tractable cognitive load: only remember things you know to be referential.
When doing non-singleton annotation, often just randomly walking document
No concrete/easy way to get shorter, hard-to-notice coreference chains.
(Unfortunately, serious studies of the differences don't seem to exist)
Utility for other layers
Some kind of document-level label is useful for everything with multi-sentence relations like set/member, part/whole, etc.
Useful if we have other things we want to encode about entities (like number)
The Argument for doing singleton annotation
We've had a request to get a bigger explanation of what we mean by "singleton" annotation and some reasons why we'd want it, so here's a case for singletons. Please post if you have disagreements/counter-proposals.
What we mean when we talk about singletons (let's focus on entities for now)
Advantages of singleton annotation (system perspective)
The General "Anaphoricity" reason
Advantages of singleton annotation (for actual representation purposes)
The general argument
The KB / Cross-document question
The "Annotation Quality" reason
Utility for other layers