search

vdobrovolskii / wl-coref

This repository contains the code for EMNLP-2021 paper "Word-Level Coreference Resolution"
MIT License
104 stars 37 forks source link

question about evaluation #30

Closed fairy-of-9 closed 2 years ago

fairy-of-9 commented 2 years ago

Hello, I have a question about evaluation. Are deleted mentions included in the evaluation?

For example, when 2% of the mention is removed, is the upper bound of the result of calcuate_conll.py 100%?

Thanks!

vdobrovolskii commented 2 years ago

Hi!

Do you mean the mentions deleted when converting spans to words?

If so, they are still deleted for word-level evaluation, but they are still there for span-level evaluation.

fairy-of-9 commented 2 years ago

In the span-level evaluation, Would it be reasonable to deduce that its impossible to get 100% if one span is removed during conversion?

Thanks!

vdobrovolskii commented 2 years ago

It would. The only spans that get deleted are the ones that have the same head words as some other spans. As the model can only predict one span per a head word, it is impossible for it to predict such spans, thus the evaluation result on CoNLL-12 can never be 1.0 on the span level

fairy-of-9 commented 2 years ago

It's clear now! Thanks for ur comments!