Investigate Claim Detection via Alignment of AMR Parses

[joecummings]

Idea:

• Parse templates into AMR Graphs
• Parse sentences from documents into AMR Graphs
• Search for similarities in the subgraphs
• If similarity between subgraph is above a certain threshold, we consider it a match

Pros:

• Can better handle variations on the templates
• Likely higher recall than RegEx matching

Cons:

• Most likely a much longer process as subgraph matching can be extremely computationally expensive

[joecummings]

Several possible notions of "similarity":

• Smatch algo
• Convert to DiGraph and use graph similarity metrics
• Semantic similarity
• Some combo of these three

[elizlee]

So far I have tried two methods:

• Comparing sentence subgraphs with template graphs using smatch -- this gives high scores to several erroneous results while also giving lower scores to many of the actual claims in the sample docs.
• Matching template root nodes and arg roles to subgraph ones -- this assumes that all instances of a given claim have the same AMR graph pattern as that of the template, but that's not always the case. Provides more accurate results than the smatch method, but still misses several claims in the text.

[joecummings]

@elizlee Just for context, can you say what the example docs are and how many of the claims they detect over the total number of expected claims?

[elizlee]

Looking at one doc made up of 27 COVID-related claims, 23 of which fall in the template categories,

• Our current method detects 20 of them
• "root matching" detects 14 of them
• Smatch only accurately matched one claim with its correct template with an F-score above 0.5, which was "X prevents COVID-19" in "No evidence exists to support taking vitamin D supplements to prevent Covid-19", but this also equally matched it with
  • X transmits COVID-19
  • X destroys COVID-19
  • X created SARS-CoV-2
  • SARS-CoV-2 was created in location/GPE/org/facility-X

[joecummings]

Diving deeper into this, what are the three claims in the template categories that we miss? Why are we missing them? @elizlee

[elizlee]

1. "coronavirus is a targeted biological weapon" i. Not sure why this one wasn't because I've seen it before...
2. "weed is a cure for coronavirus" ii. It doesn't quite match the "X cures COVID-19" pattern
3. "No evidence exists to support taking vitamin D supplements to prevent Covid-19" iii. While the claim here would be "taking vitamin D supplements prevents Covid-19," in the sentence this has a different structure from "X prevents COVID-19"

[joecummings]

What does the AMR parse look like between "Weed is a cure for coronavirus" and "X cures coronavirus"?

[elizlee]

The portion of the sentence graph with the claim:

:ARG1 (c2 / cure-01
            :ARG1 (c3 / coronavirus)
            :ARG2 (w / weed)
            :example (c4 / cannabis)))

The "X cures COVID-19" graph:

(c / cure-01
      :ARG0 (s / something)
      :ARG1 (p / person
            :name (n / name
                  :op1 "COVID-19")))

The PB argument roles for cure-01 are supposed to be:

ARG0: doctor (etc)
ARG1: patient
ARG2: illness or injury
ARG3: instrument, treatment

So actually, both have their arguments in the wrong slots. (Though, I can see why an "X cures Y" structure would make it think that Y is the "patient" and not the disease)