May 9: edited to reflect current state of MS-AMR discussion
Let's talk about metrics!
I'm going to propose a simple SMATCH-like thing for coreference. I don't have a system that does this yet but provided mock-ups files that should simulate it using the normal SMATCH code.
Mainly, I'd propose that we actually merge coreferent nodes, and remove fully redundant links. I'd propose that otherwise we just bind all the AMRs together with a "multi-sentence" label.
For entities, this is simple. For example, if you have:
Merging coreferent events is more complicated. The complicated thing about this is the numbered arguments, which are secretly dependent on the predicate type; if we merge nodes, they suddenly have many types, and so that assumption breaks. Conceptually I'd think about it this way:
(a / discourse-variable
:instance-of "attack-01"
:instance-of "assault-01"
:instance-of "beat-01"
:ARG0(attack-01) y
:ARG0(assault-01) y
:ARG0(beat-01) y
:ARG1(attack-01) p
:ARG1(assault-01) p
:ARG1(beat-01) p
:manner (v / vicious)
:ARG1-of (c / cause-01
:ARG0 (r / race)))
A vs B gets (0.91)
A vs C gets (0.81)
A vs D gets (0.73)
Thoughts? Alternative proposals? I'd suggest that the big question is whether we want to treat every coreferent variable as actually all being the same point in the graph, and I think it's cleaner it we do.
Multisentence SMATCH
May 9: edited to reflect current state of MS-AMR discussion
Let's talk about metrics!
I'm going to propose a simple SMATCH-like thing for coreference. I don't have a system that does this yet but provided mock-ups files that should simulate it using the normal SMATCH code.
Mainly, I'd propose that we actually merge coreferent nodes, and remove fully redundant links. I'd propose that otherwise we just bind all the AMRs together with a "multi-sentence" label.
For entities, this is simple. For example, if you have:
Predicates
Merging coreferent events is more complicated. The complicated thing about this is the numbered arguments, which are secretly dependent on the predicate type; if we merge nodes, they suddenly have many types, and so that assumption breaks. Conceptually I'd think about it this way:
Ideally this could be done over normally releases of AMRs with metadata, but for feeding into SMATCH right now, it requires transforming things so that you don't have repeated variables. I've put together four mock documents for what I'd propose SMATCH should look at: A: A document with the annotations we've provided for entity/event coreference- roughly, Tim's annotations B: A second document with the kind of variance we saw in our annotations - roughly, Jeff's annotations C: The LDC annotation of the same document D: The LDC annotation of the same document, but with a big coreference error (merging the two main coreference clusters)
A vs B gets (0.91) A vs C gets (0.81) A vs D gets (0.73)
Thoughts? Alternative proposals? I'd suggest that the big question is whether we want to treat every coreferent variable as actually all being the same point in the graph, and I think it's cleaner it we do.