search

lephong / mulrel-nel

named entity linking with latent relations
Apache License 2.0
172 stars 36 forks source link

About the F1 metric #23

Closed jianyucai closed 4 years ago

jianyucai commented 4 years ago

Hi, thanks for your code. I noticed that you choose F1 as the evaluation metric, but I am a little confused on its definition.

https://github.com/lephong/mulrel-nel/blob/db14942450f72c87a4d46349860e96ef2edf353d/nel/dataset.py#L187-L203

We know that precision and recall are defined by: precision = tp / (tp + fp) recall = tp / (tp + fn)

However, this task is not a binary classification task. So I'm wondering how to define tp, tn, fp and fn.

lephong commented 4 years ago

It's pretty straight-forward to extend precision/recall to multi-class classification. In this case, true positive = the number of predicted y_i = golden y_i.

jianyucai commented 4 years ago

Thanks.