Score the performance

[lcreteig]

Evaluate the performance of the rules as in the ContextD paper (precision, recall, F1-score, separate for each document type).

For each labeled entity in a doc (for ent in doc.ents):

• TP: ent.label is 'NEGATED'; ent._.is_negated (medspacy) is True
• FN: ent.label is 'NEGATED'; ent._.is_negated (medspacy) is False
• FP: ent.label is 'NOT_NEGATED'; ent._.is_negated (medspacy) is True
• TN: ent.label is 'NOT_NEGATED'; ent._.is_negated (medspacy) is False

Note that the other approaches use some part of the ECC dataset for training, so coordinate with them which split to actually evaluate on

[lcreteig]

Make this congruent with the general evaluation code, so that we can join the predictions from all models for each entity:

• [x] Add entity_id to each annotation: {DOCTYPE}_{START}_{STOP} (e.g. DL1111_33_38)

Right now most of the rule-based "pipeline" occurs in the preprocessing step. Could also consider matching the pipeline design of the other models, where the raw text is loaded in to make a prediction. In that case, everything would happen in one pipeline, including tokenisation, labelling, and sentence splitting.

• Can make a "labeller" component, and add start/end "arguments" to nlp: https://spacy.io/api/language#call, https://github.com/explosion/spaCy/issues/2740. That would enable running the entire pipeline on one text to annotate one entity.