make root in AMR formalism anonymous or non-anonymous?

[flipz357]

Currently, the root in the AMR formalism of SMATCH++ is modeled as a relation from the root variable to the concept of the root node (the focus), with a ":root" relation, e.g.:

(b / boy)
triples = [(b, :root, boy), (b, :instance, boy)]

We call this "non-anonymous" root, since the root directly holds the explicit concept/focus of the sentence (boy).

Another option, however, is

(b / boy)
triples = [(ROOT, :root, b), (b, :instance, boy)]

This option is an ""anonymous" root. It is already used in SMATCH++ for generic graphs other than AMR.

There are implications of the two options for Matching/IAA/Similarity that have triggered extended discussion with @goodmami and @jheinecke in this longer thread 1.

As per the latest update of the AMR guidelines from @nscheid, it seems that the anonymous form may be preferred.

To summarize, what is the advantage of setting the anonymous root in SMATCH++ for AMR formalism?

• It gives higher scores (this is not really an advantage, but something that folks may like...).
• Then there are a few differences that could be desirable, as @goodmami pointed out here.
• It seems to (now) better align with AMR guidelines

What is the disadvantage of setting the anonymous root in SMATCH++ for AMR formalism?

• The correlation to human similarity is lower on AMR similarity rating
• There are then awkward cases like:

(b / boy)

having a (unreasonably high) similarity of 0.5 to

(r / random-concept)

These cases happen because the anonymous root can be matched (as happens in snowblink14/smatch).

Don't know what to do at this point. Either of the two options seem ok but not ideal for different reasons.

Any help/advice is appreciated.

[jheinecke]

Thanks for sharing your thoughts on this. I had a quick idea to avoid the 0.5 score four your example above. Smatch (any flavour) could ignore the matching root node in the evaluation as long as it is the only matching node. In this case

( b / boy)

and

(d /dog)

but for

(s / sleep-01
   :ARG0 (d / dog))

and

(d / dog
  :ARG0-of (s / sleep))

the different topic/focus is taken into account. I did not have yet the time to think through all the implications of this. How should score the following couples a / b and a /c

a)

(s / sleep-01
  :ARG0 (b / boy))

b)

(s / snore-01
  :ARG0 (d / dog))

c)

(d / dog)
  :ARG0-of (s / snore-01))

[flipz357]

I like your approach of thinking! Just:

ignore the matching root node in the evaluation as long as it is the only matching node

This sounds a bit like a post-processing step (doing an if/else after the ILP has found solution), and that can lead to non-trivial problems. You can see this when considering cases with two global optimal solution. E.g., for a) and c) you can align s=d; b=s, which gives one match (the root will be matching ROOT :root s=d), or s=s and b=d (also giving one match, the s=s :arg0 b=d). Then depending on which solution is selected, the graphs may receive a score of zero, or not zero. So this is not so good.

Therefore it may be probably be best if the issue can be handled in the pre-processing/graph standardizing stage, before the alignment. Maybe your proposal or something similar to it can work here too, but I am not sure.

[flipz357]

Just for reference, linking this issue in the AMR guidelines.

The "root" vs "focus" equivalence/distinction might be not so clear even in the AMR guidelines.

[flipz357]

Happy to report that I consider this issue is fixed in v1.7.0! 🎉

Solution:

• As default, Smatch now uses -graph_type generic which uses anonymous root.

• The user can also select -graph_type amr which uses non-anonymous root for stricter AMR-tailored matching.

• Importantly, I added warning in the README about input format. For full user-control over how the graph is read and how the root is treated, the input format needs be set to tsv. See the note here.

Indeed, I now have come to believe that the root "issue" is as much an issue of the Penman format as it is of the AMR formalism. This is because it is generally underspecified what the root is/what it represents, not only for AMR, but also for Penman. To achieve full control over how the graph looks, tsv format must be used from the command line, or the triples should be fed directly via the Python interface (an example for the latter is Ex VI).

I think this is an elegant and correct solution. But correct me if I overlooked something.