search

fenchri / edge-oriented-graph

Source code for the EMNLP 2019 paper: "Connecting the Dots: Document-level Relation Extraction with Edge-oriented Graphs"
Other
146 stars 17 forks source link

Question about result evaluation #11

Closed light8lee closed 4 years ago

light8lee commented 4 years ago

why the gold data you generated in CDR dataset contains 1:NR:2? I think it should only consider about 1:CID:2 type. For example, you generated follow lines as gold data

8701013|D015738|D003693|NON-CROSS|1:CID:2
8701013|D015738|D014456|NON-CROSS|1:NR:2
439781|D007213|D007022|NON-CROSS|1:CID:2
439781|D012964|D007022|NON-CROSS|1:NR:2
439781|D011453|D007022|CROSS|1:NR:2
439781|D000809|D007022|CROSS|1:NR:2

why not

8701013|D015738|D003693|NON-CROSS|1:CID:2
439781|D007213|D007022|NON-CROSS|1:CID:2
fenchri commented 4 years ago

During evaluation I consider only the positive pairs, but I generated this for some analysis I did. If you check the evaluation.py script you will see that the NR pairs are actually skipped.

BTW, the script works for binary classification problems, I will make it generalisable soon.

light8lee commented 4 years ago

OK, I find that part, you are right.