Improvements to NEL evaluation

[wricketts]

New Feature Summary

Update nel/evaluate.py to make use of this package (if feasible).

Related

No response

Alternatives

No response

Additional context

Github page

[wricketts]

Use of neleval package deemed not feasible (broken links in the documentation make it hard to follow, plus it requires a format that our data is not in).

Instead, the plan is to modify evaluate.py following the description used in this paper.

Evaluation treats each linking annotation as a tuple (Span, type, kbid).

• Span consists of the document ID (for our purposes, the GUID) and the begin/end character offsets of the entity in the text.
• Type corresponds to Category (one of Person, Location, Organization, Product, Event and Title).
• KBID (Knowledge Base ID) for our use case is the Wikidata Q Identifier.

Metrics are calculated as follows:

• Precision: computed by taking the cardinality of the intersection between the set of Gold annotations and the set of System annotations, divided by the cardinality of the set of System annotations.
• Recall: computed by taking the cardinality of the intersection between the set of Gold annotations and the set of System annotations, divided by the cardinality of the set of Gold annotations.
• F1: The conventional F1 formula.

The dbpedia spotlight app does not seem to perform NIL recognition (marking entities as not having a node in a KB), so this will not be evaluated.