Utilizing strictly semantic similarity (PhraseBERT or SentenceBERT w/ interesting chunking)
Frame as LM paraphrasing task - if a sentence could be "paraphrased" into one of the provided topics, then it might be a claim.
Better AMR (seems like a losing battle, but worth spending a little bit of time trying to see if there's a better AMR parser out there)
Graph + SS approach - Encode an AMR parse as an actual graph and use graph similarity metrics plus some token semantic similarity to determine if a chunk is similar to one of the claim templates.
Dec 2
Claimbuster + AMR/FrameNet to find claim span
Add custom covid claim frames to FrameNet and use existing software to match Frames in the wild
Other random (not so great) ideas:
If a sentence has a VBD and numbers, consider it a claim. Would certainly have a higher recall.
Taking the cosine similarity between an encoded sentence and each encoded template should tell us which (if any) of the templates it is most similar to.
Reasoning:
This has worked fairly well for many other of our purposes. Templates and claims should be much more semantically similar than other text spans.
Loose implementation:
Break sentences into clauses.
Encode clauses with either SentenceBERT or PhraseBERT
Encode provided claim templates with whichever model was chosen for 2
Do a pairwise cosine similarity measurement between all clauses and all templates using SBERT's utils
Using the ~50 positive claims we already have, see what the cosine similarity is between those clauses and the correct template
Average the cosine similarity scores for these pairs and have that be our threshold for determining if a clause is a claim.
Measuring
How many of the ~50 positive claims can this method find?
AMR graphs closely enough resemble standard graphs that we can use graph similarity metrics.
Graph similiarity will sometimes not be enough, and we can rely on encoding noun phrases to determine similiary, as well.
Reasoning:
SMatch and other AMR comparison metrics don't seem well suited for determining similarity. Although variations in sentence structure can produce slightly different AMR graphs, the differences will be smaller than those of a completed different AMR parse. Therefore, standard graph metrics might be a good way to capture these small changes.
Loose implementation:
Parse every sentence into an AMR graph.
Convert AMR graphs into NetworkX graphs.
Parse every template into an AMR graph.
Convert template graphs into NetworkX graphs.
Calculate a similarity score between every AMR graph and template graph.
See lestat-alignment for some choices of similarity scores
Do all steps in Utilizing semantic similarity for getting a SS score.
Combine graph sim scores and semantic sim scores.
Measuring
How many of the ~50 positive claims can this method find?
How many extra claims does this method find?
How many bad examples does this method find?
How long does it take to do this entire process?
Is the graph method or the semantic sim method more important?
Claimbuster is good enough to detect claim sentences, but not claim spans.
The majority of claim clauses follow the patterns we identified through FrameNet.
Reasoning:
In a spot check of UIUC's output (using Claimbuster), it was fairly good at detecting sentences which contained claims, but it was pretty bad at detecting what part of that sentence contained a claim. In our experiments for determining the "claimer", we actually build a tool that can detect which part of an AMR graph represents a claim.
Loose implementation:
Run Claimbuster over every sentence in the corpora.
Parse every sentence that Claimbuster returns as a probable claim into an AMR graph.
Extract the subgraph in the AMR graph for the verbs that we already identified as likely to be referring to a claim.
Reconstruct the claim span from the AMR subgraph.
Measuring:
How many of the ~50 positive claims can this method find?
How many extra claims does this method find?
How many bad examples does this method find?
How long does it take to do this entire process?
What is the split between finding good sentences vs finding good clauses?
We can create paraphrases from claim sentences that are more easily comparable to claim topics.
Reasoning:
Given that semantic similarity has already worked fairly well, comparing paraphrased versions of each clause would in theory yield more accurate similarity scores.
Measure semantic similarity using cosine over SpaCy vectors
Look at ~50 claims we already have and compare cosine similarity between those clauses and the correct template
Average the cosine similarity scores for these pairs and have that be our threshold for determining if a clause is a claim.
Measuring:
How many of the ~50 positive claims can this method find?
How many extra claims does this method find?
How many bad examples does this method find?
How long does it take to do this entire process?
Cons:
From early testing of the paraphrase generator mentioned, it doesn't actually appear to do much paraphrasing, and if this trend persists, it might as well be plan 1.
There exists a better-performing AMR parser than IBM's Transition AMR Parser that produces output which better matches the claim subgraphs with claim topic template graphs.
Reasoning:
A system that yields more accurate AMR graphs may yield more accurate semantic similarity scores between the claim graph and a template graph.
Loose implementation:
Replace Transition AMR Parser with a higher-performing parser and run it on all sentences including the claim topic templates
Run AMR graph semantic similarity metric on each subgraph against each claim template
Look at ~50 claims we already have and compare cosine similarity between those clauses and the correct template
Average the cosine similarity scores for these pairs and have that be our threshold for determining if a clause is a claim.
Measuring:
How many of the ~50 positive claims can this method find?
How many extra claims does this method find?
How many bad examples does this method find?
How long does it take to do this entire process?
How does this compare with the results from the Transition AMR Parser?
Cons:
IIRC, we selected the Transition AMR Parser because it was one of the leading ones out there. Although there are others with slightly better Smatch, I doubt that replacing our current parser with those would provide much of an improvement boost, and time would be better spent trying an alternate method.
Side note:
https://github.com/SapienzaNLP/spring at least appears to have a simpler setup process (although it would still need its own environment). If time, perhaps it's worth trying out just to see if we can make installation less flaky.
Initial Brainstorm Joe & Liz (Dec 1):
Dec 2
Other random (not so great) ideas: