search

bekou / multihead_joint_entity_relation_extraction

Implementation of our papers Joint entity recognition and relation extraction as a multi-head selection problem (Expert Syst. Appl, 2018) and Adversarial training for multi-context joint entity and relation extraction (EMNLP, 2018).
MIT License
413 stars 112 forks source link

Evaluation: relation F1 #17

Closed WindChimeRan closed 5 years ago

WindChimeRan commented 5 years ago

Hi,

I am confused about the relation F1 in the paper. For me, a triplet is correct only if the relation and all the tokens of head and tail are correct, eg: (B-PERSON √, I-PERSON √, Born_in √, B-LOCATION √, I-LOCATION√), I will count it as correct_triplet += 1

Is it the relation F1 in your EMNLP paper? You said CoNLL04 baseline+AT Relation F1 = 61.95

Thank you

bekou commented 5 years ago

Yes in the strict evaluation (i.e., 61.95), I count it like this (i.e., both entities and relation should be correct). From the paper: "S(trict): we score an entity as correct if both the entity boundaries and the entity type are correct (ACE04, ADE, CoNLL04, DREC). In all cases, a relation is considered as correct when both the relation type and the argument entities are correct."