search

MantisAI / nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13
MIT License
154 stars 19 forks source link

Why aren't tokens matched? #53

Closed vedantc6 closed 1 year ago

vedantc6 commented 3 years ago

The question might be stupid, but I see there's no actual word given as input in the Prodigy span style. For an example:

Truth: "Paris" -> [[{"label": "PER", "start": 0, "end": 1}]] Pred: "London" -> [[{"label": "PER", "start": 0, "end": 1}]]

The evaluation shouldn't have Precision, Recall and F1 as 1.0 but it does. Even if I change the format to conll style, it is of no use.

Am I missing something?

image

ivyleavedtoadflax commented 2 years ago

Hi @vedantc6, thanks for raising an issue. I think you are using the tool for a use case that it was not intended to meet. The tool is for comparing sets of annotations on the same input document. It simply will not function when the input documents are different (as you have found).

ivyleavedtoadflax commented 1 year ago

Assuming this is stale, and closing.