weighting of partial matches in the fuzzy evaluation scoring

hipe-eval / HIPE-scorer

A python module for evaluating NERC and NEL system performances as defined in the HIPE shared tasks (formerly CLEF-HIPE-2020-scorer).

https://hipe-eval.github.io

MIT License

13 stars 4 forks source link

weighting of partial matches in the fuzzy evaluation scoring #4

Closed aflueckiger closed 4 years ago

aflueckiger commented 4 years ago

@e-maud @mromanello Do we reward partial matches as high as full matches (i.e. exact boundaries) in our fuzzy evaluation scoring? This would mean setting the formula the 0.5 to 1 in the formula below.

DeepinScreenshot_select-area_20200130162256

Source: http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/

simon-clematide commented 4 years ago

My take on it is that for fuzzy evaluation we are really relaxed, meaning setting it to 1. But your mileage might vary.

e-maud commented 4 years ago

Idem, I would be very relaxed for fuzzy: one token overlap suffices to get full score, and very strict for exact: boundary should match exactly. So yes, 1 for partial matches in fuzzy setting.

mromanello commented 4 years ago

Fine by me as well!

aflueckiger commented 4 years ago

Ok, our implementation follows this proposal.