MichaelRoeder
commented
6 years ago Thanks for that question. I can only give a general answer since you have uploaded a larger dataset. I think uploading an example with a single document for which the evaluation results are different would give use an easier way of comparing the evaluations :wink:
In general, RT2KB does the following:
- it identifies entities that have been recognized correctly (Recognition step)
- from these correctly identified entities it takes the types and calculates the hierarchical F-measure for the type. (Errors in the recognition will lead to lower precision/recall in this calculation as well, since expected type information won't be available, etc.)
From the results for these two single steps, you can see that the benchmarked system has got 0.76 F1-measure for each step. So the combination of both can not get more than that and most probably will have a lower F1-measure since correctly identified entities might have got a (partly) wrong type.
However, I would be happy to dig into this when you can provide a single example with the results from the other scorers :smiley:
jplu
rtroncy
TortugaAttack
Hello,
The RT2KB and Typing process gives strange scores compared to other scorers. Every time I run a RT2KB process, on a NIF dataset, I always get the exact same score for Precision, Recall and F1, which is quite odd (see this example). If I evaluate the same output with two other scorers (neleval and conlleval) I get the same results with both scorers that are much higher than what RT2KB can gives me (P = 0.717, R = 0.765, F1 = 0.740).
The description of RT2KB says "the annotator gets a text and shall recognize the entities inside and their types", consequently I'm curious to know how the three measures can be equals for Typing when they are different for Recognition.
Any light on this would be welcomed :)
Thanks!