search

wikilinks / neleval

Entity disambiguation evaluation and error analysis tool
Apache License 2.0
116 stars 23 forks source link

prepare-tac should merge multiple candidates at same location #11

Closed jnothman closed 9 years ago

jnothman commented 9 years ago

I've hacked this into Reader/Document at https://github.com/wikilinks/neleval/compare/merge-duplicates?expand=1, but given that there is a facility to read in multiple candidates from each line of a .tsv, this should really be happening in prepare-tac.

But there should still be an error raised if the gold data for prepare-tac has multiple candidates for any annotation. So prepare-tac should probably have a --gold mode.

benhachey commented 9 years ago

Agreed.

In fact, validation should probably be task-specific as well. Allowability of embedded or crossing mentions varies by task, as can slice semantics.

jnothman commented 9 years ago

Well that handling is all configurable in the current implementation, it just lacks a way to be specified from the command-line.

On 20 November 2014 11:24, Ben Hachey notifications@github.com wrote:

Agreed.

In fact, validation should probably be task-specific as well. Allowability of embedded or crossing mentions varies by task, as can slice semantics.

— Reply to this email directly or view it on GitHub https://github.com/wikilinks/neleval/issues/11#issuecomment-63740786.

jnothman commented 9 years ago

Currently the hacky version of this functionality is in a branch, so results reported by master are incorrect. We need to resolve this ASAP.

jnothman commented 9 years ago

This was fixed by #13.