search

PlusLabNLP / DEGREE

Code for our NAACL-2022 paper DEGREE: A Data-Efficient Generation-Based Event Extraction Model.
Apache License 2.0
74 stars 12 forks source link

A question about test set #9

Closed humoxian closed 1 year ago

humoxian commented 1 year ago

Hello! As the paper said, DEGREE(ED) must run once for each event type in the target ontology, since it only considers one event type at a time. But I noticed that in 'test_all_pkl', it still remain randomly selecting m(m=15 in ACE dataset ) event types as queries for each example just like training. image I want to reappear your code on ACE2005, but found my F1-score was very low(usually predicting many irrelevant event types). So I compared my data form with yours and then found this question. Looking forward to your reply.Thanks!

ej0cl6 commented 1 year ago

Hi, thanks for your interest in our work. During training, we set the number of negative examples to 15 (you can check the config). However, when evaluating, we do consider all the event types. Please use follow https://github.com/PlusLabNLP/DEGREE#evaluation and use either eval_end2endEE.py or eval_pipelineEE.py to consider all types for evaluation.