search

nickyringland / nested_named_entities

63 stars 7 forks source link

How to evaluate the performance #3

Closed thtb closed 5 years ago

thtb commented 5 years ago

There are cases that a span is annotated with more than one entities. How to calculate the precision and recall of these spans. Could you give an official performance calculation script given a formatted predictions.

dainlp commented 5 years ago

If one span is annotated with multiple entity types, then it will be treated as multiple mentions. Our evaluation is on mention-level that the prediction is correct only if the span boundary and the entity type are all the same as the gold annotation.

I also uploaded the outputs from our flat BiLSTM-CRF models which use either the outermost or the innermost mentions for training, and the code used to produce BiLSTM-CRF-BOTH (combine the outputs from these two flat models) result in Table 2 of our paper. Hope it helps.

thtb commented 5 years ago

It helps a lot.