search

cfmrp / mtool

Software to Manipulate Different Flavors of Semantic Graphs
http://mrp.nlpl.eu
GNU Lesser General Public License v3.0
51 stars 24 forks source link

Case-sensitve label scoring in MRP metric #41

Closed namednil closed 5 years ago

namednil commented 5 years ago

According to the description, case shouldn't matter for comparison of node labels. However, I get different results for:

{"id": "20034001", "flavor": 0, "framework": "psd", "version": 0.9, "time": "2019-04-10 (20:25)", "input": "Pick a country, any country.", "tops": [0], "nodes": [{"id": 0, "label": "pick", "properties": ["pos", "frame"], "values": ["VB", "ev-w2265f5"], "anchors": [{"from": 0, "to": 4}]}, {"id": 2, "label": "country", "properties": ["pos"], "values": ["NN"], "anchors": [{"from": 7, "to": 14}]}, {"id": 3, "label": "#Comma", "properties": ["pos"], "values": [","], "anchors": [{"from": 14, "to": 15}]}, {"id": 4, "label": "any", "properties": ["pos"], "values": ["DT"], "anchors": [{"from": 16, "to": 19}]}, {"id": 5, "label": "country", "properties": ["pos"], "values": ["NN"], "anchors": [{"from": 20, "to": 27}]}], "edges": [{"source": 3, "target": 0, "label": "CONJ.member"}, {"source": 0, "target": 2, "label": "PAT-arg"}, {"source": 5, "target": 4, "label": "RSTR"}, {"source": 0, "target": 5, "label": "PAT-arg"}]}

and

{"id": "20034001", "flavor": 0, "framework": "psd", "version": 0.9, "time": "2019-04-10 (20:25)", "input": "Pick a country, any country.", "tops": [0], "nodes": [{"id": 0, "label": "pick", "properties": ["pos", "frame"], "values": ["VB", "ev-w2265f5"], "anchors": [{"from": 0, "to": 4}]}, {"id": 2, "label": "country", "properties": ["pos"], "values": ["NN"], "anchors": [{"from": 7, "to": 14}]}, {"id": 3, "label": "#comma", "properties": ["pos"], "values": [","], "anchors": [{"from": 14, "to": 15}]}, {"id": 4, "label": "any", "properties": ["pos"], "values": ["DT"], "anchors": [{"from": 16, "to": 19}]}, {"id": 5, "label": "country", "properties": ["pos"], "values": ["NN"], "anchors": [{"from": 20, "to": 27}]}], "edges": [{"source": 3, "target": 0, "label": "CONJ.member"}, {"source": 0, "target": 2, "label": "PAT-arg"}, {"source": 5, "target": 4, "label": "RSTR"}, {"source": 0, "target": 5, "label": "PAT-arg"}]}

The only difference between the two is #Comma vs #comma. I used the current version (commit c1f809e63c5921188eab80841f13f791d9f1f73e) with the option --score mrp.

oepen commented 5 years ago

thanks for alerting us to this one! the string normalization described on the task web site had yet to be implemented; done now :-).