Smatch (still) yields scores above 100%

ramon-astudillo commented 4 years ago

This seems like a bug similar to the solved

https://github.com/snowblink14/smatch/issues/15

which can cause Smatch scores to be artificially high by double counting edges and eventually reach values above 100%. The only way of knowing if a Smatch score below 100% is real or suffers from this bug is computing the Smatch of a file with itself (henceforce self-Smatch).

An example

# ::tok Uh ... Do you have legislative power or enforcement power ? <ROOT>
(h / have-03
      :ARG0 (y / you)
      :ARG1 (o / or
            :op1 (p / power
                  :instrument-of (l / legislate-01))
            :op2 (p2 / power
                  :instrument-of (e / enforce-01)))
      :mod (u / uh
            :mode expressive)
      :mode interrogative)

has the expected self-Smatch of 100%, while

# ::tok Uh ... Do you have legislative power or enforcement power ? <ROOT>
(h / have-03
      :ARG0 (y / you)
      :ARG1 (o / or
            :op1 (p / power
                  :instrument-of (l / legislate-01))
            :op2 (p2 / power
                  :instrument-of (e / enforce-01)))
      :mod (u / uh
            :mode expressive)
      :mode interrogative
      :mode interrogative)

has self-Smatch 110.5% due to the repeated :mode interrogative. The score can grow ad-infinitum just by repeating further.

goodmami commented 4 years ago

This is a duplicate of #15, which is not actually "solved" (you may have seen the red "closed" graphic at the bottom of the issue, but note that that is for a linked issue in mtool and not for smatch).

But you're right that this is a real issue that needs to be addressed. I suggest closing this issue and continuing the conversation at #15.

ramon-astudillo commented 4 years ago

thanks moving discussion to https://github.com/snowblink14/smatch/issues/15

snowblink14 / smatch

Smatch (still) yields scores above 100% #28