TOP problem

[rafaelanchieta]

Considering TOP as an attribute as suggested by @oepen in #25, Smatch is returning high scores for completely different nodes.

$ cat x
(x / x)

$ cat y
(y / y)

python smatch.py --pr -f x y 
Precision: 0.5
Recall: 0.5
F-score: 0.5

[oepen]

well, one might quarrel whether the two nodes are 'completely different'. i find it hard to have linguistic intuitions about this example, but formally i see two graphs that are somewhat similar: they are both comprised of exactly one node and have no edges. the only difference between the two graphs is that the label for the node differs, so i am on board with SMATCH here: of the observable information in the two graphs, half of it is shared :-).

[rafaelanchieta]

I agree that they are structurally similar. However, the minimal Smatch score will be 0.5.

$ cat x
(d \ die-01)

$ cat y
(r \ run-01)

python smatch.py --pr - f x y
Precision: 0.5
Recall: 0.5
F-score: 0.5

[goodmami]

the minimal Smatch score will be 0.5.

Only if both sides have only 2 triples (TOP and instance) as in this example. Increase the number of triples and the score can go lower.

$ cat a
(d / die-01
   :ARG0 (p / process))
$ cat b
(r / run-01)
$ python3 smatch.py --pr -f a b 
Precision: 0.25
Recall: 0.50
F-score: 0.33

[rafaelanchieta]

You are right. The score may decrease. Is that the expected behaviour of Smatch?

[goodmami]

Is that the expected behaviour of Smatch?

Yes. The more different the graphs, the lower the score.

[rafaelanchieta]

Ok. Thanks.