search

snowblink14 / smatch

Smatch tool: evaluation of AMR semantic structures
MIT License
63 stars 27 forks source link

Issue about the 'TOP' attribute relation #18

Open jinxing94 opened 5 years ago

jinxing94 commented 5 years ago

Hi,

I pass two AMR strings with same meaning but do not get score 1. The only difference between the two strings is that one has ARG2 and another has ARG2-of. And I find that this result in different "TOP" attribute relation and thus the computed smatch score is not 1. I am wondering why the "TOP" attribute relation should be added and how to fix this problem.

Below are the two strings:

(e / except-01 :ARG2 (c/ change-01 :ARG1 (n/ nothing)) :ARG1 (p / pass-01 :ARG2 (l / law :name (n2 / name :op1 "Obaminationcare") :wiki "Patient_Protection_and_Affordable_Care_Act")))

(c / change-01:ARG1 (n / nothing):ARG2-of (e / except-01:ARG1 (p / pass-01:ARG2 (l / law :wiki "Patient_Protection_and_Affordable_Care_Act":name (n2 / name :op1 "Obaminationcare")))))

I use AMR.parse_AMR_line to parse the above two strings and get the following triples:

instance triple ('instance', 'e', 'except-01') ('instance', 'c', 'change-01') ('instance', 'n', 'nothing') ('instance', 'p', 'pass-01') ('instance', 'l', 'law') ('instance', 'n2', 'name') attribute triple ('TOP', 'e', 'except-01') ('wiki', 'l', 'Patient_Protection_and_Affordable_CareAct') ('op1', 'n2', 'Obaminationcare_') relation triple ('ARG2', 'e', 'c') ('ARG1', 'e', 'p') ('ARG1', 'c', 'n') ('ARG2', 'p', 'l') ('name', 'l', 'n2')

instance triple ('instance', 'c', 'change-01') ('instance', 'n', 'nothing') ('instance', 'e', 'except-01') ('instance', 'p', 'pass-01') ('instance', 'l', 'law') ('instance', 'n2', 'name') attribute triple ('TOP', 'c', 'change-01') ('wiki', 'l', 'Patient_Protection_and_Affordable_CareAct') ('op1', 'n2', 'Obaminationcare_') relation triple ('ARG1', 'c', 'n') ('ARG2', 'e', 'c') ('ARG1', 'e', 'p') ('ARG2', 'p', 'l') ('name', 'l', 'n2')

miguelballesteros commented 5 years ago

See issue #15

jinxing94 commented 5 years ago

Hi,

I think my issue is not the same as yours. There is no duplication in my example.

nschneid commented 5 years ago

TOP (or the "focus") refers to the concept that is at the top at the AMR. It is except-01 in your first example and change-01 in your second example. So these are not identical AMRs. Typically what should be the TOP concept depends on the main verb of the sentence.

jinxing94 commented 5 years ago

Got it. Thanks.

rafaelanchieta commented 5 years ago

Recently, a new metric has been published. https://github.com/rafaelanchieta/sema

nschneid commented 5 years ago

Recently, a new metric has been published.

@rafaelanchieta Could you please provide a link to a description of the metric? How does it differ from smatch?

miguelballesteros commented 5 years ago

It would be good to know how some parsers compare with this metric. Thanks.

rafaelanchieta commented 5 years ago

http://143.107.183.175:23580/sema/static/metric/preprint-cicling.pdf

If there is any question, please let me know.

goodmami commented 4 years ago

As @nschneid said above:

TOP (or the "focus") refers to the concept that is at the top at the AMR. It is except-01 in your first example and change-01 in your second example. So these are not identical AMRs. Typically what should be the TOP concept depends on the main verb of the sentence.

I think there is nothing to do here and the issue should be closed.