swabhs / open-sesame

A frame-semantic parsing system based on a softmax-margin SegRNN.
Apache License 2.0
229 stars 65 forks source link

Understanding the output #44

Closed free-soellingeraj closed 4 years ago

free-soellingeraj commented 4 years ago

For each sentence, the argid model is producing 2 conll2009 matrices that can contain conflicting values for ROLE variable. Can someone explain why there are 2 conll2009 entries for each sentence? Why would they be in conflict with each other? See below example:

1   the _   The _   DT  0   _   _   _   _   _   _   _   O
2   stock   _   stock   _   NN  0   _   _   _   _   _   stock.n Store   S-Supply
3   was _   be  _   VBD 0   _   _   _   _   _   _   _   O
4   bought  _   buy _   VBN 0   _   _   _   _   _   _   _   O
5   by  _   by  _   IN  0   _   _   _   _   _   _   _   O
6   bob _   Bob _   NNP 0   _   _   _   _   _   _   _   O
7   .   _   .   _   .   0   _   _   _   _   _   _   _   O

1   the _   The _   DT  0   _   _   _   _   _   _   _   B-Goods
2   stock   _   stock   _   NN  0   _   _   _   _   _   _   _   I-Goods
3   was _   be  _   VBD 0   _   _   _   _   _   _   _   O
4   bought  _   buy _   VBN 0   _   _   _   _   _   buy.v   Commerce_buy    O
5   by  _   by  _   IN  0   _   _   _   _   _   _   _   B-Goods
6   bob _   Bob _   NNP 0   _   _   _   _   _   _   _   I-Goods
7   .   _   .   _   .   0   _   _   _   _   _   _   _   O

1   bob _   Bob _   NNP 1   _   _   _   _   _   _   _   S-Goods
2   bought  _   buy _   VBD 1   _   _   _   _   _   buy.v   Commerce_buy    O
3   the _   the _   DT  1   _   _   _   _   _   _   _   B-Goods
4   stock   _   stock   _   NN  1   _   _   _   _   _   _   _   I-Goods
5   .   _   .   _   .   1   _   _   _   _   _   _   _   O

1   bob _   Bob _   NNP 1   _   _   _   _   _   _   _   O
2   bought  _   buy _   VBD 1   _   _   _   _   _   _   _   O
3   the _   the _   DT  1   _   _   _   _   _   _   _   O
4   stock   _   stock   _   NN  1   _   _   _   _   _   stock.n Store   S-Supply
5   .   _   .   _   .   1   _   _   _   _   _   _   _   O
swabhs commented 4 years ago

These are not in conflict, instead they are showing role labeling structures for different frames. If you look at the last but one column, you can see the frame for which the arguments are being produces. Hope this helps!

free-soellingeraj commented 4 years ago

I am still confused about how to interpret this little experiment I did (above). What I am looking for is confirmation that these sentences are highly similar to each other in the information that they convey. I used the same words in this case, but what if I swap out synonyms like converting stock -> inventory? How do you personally interpret this little example above using the output shown above? Thanks in advance.

swabhs commented 4 years ago

As far as I can tell, the frames and role structures look identical. As for the difference in voice (active vs. passive) the correct arguments are "Bob" and "by Bob" respectively, based on the FrameNet annotation scheme. The predicted argument labels are incorrect in the attribution of "Bob" as Goods instead of Buyer. But the predictions for the two sentences look consistent to me.