Using doc.spans["sc"] (SpanCategorizer) to solve the problem of overlapped tokens in nested NER for spacy. By replacing doc.ents with doc.spans["sc"], all possible entities are able to be stored without any errors.
After storing all possible spans, we filter out overlapping spans before adding them to doc.ents. Here we remove overlapping spans using spacy.util.filter_spans. When spans overlap, the rule is to prefer the first longest span over shorter ones.
Using
doc.spans["sc"](SpanCategorizer) to solve the problem of overlapped tokens in nested NER for spacy. By replacingdoc.entswithdoc.spans["sc"], all possible entities are able to be stored without any errors. After storing all possible spans, we filter out overlapping spans before adding them todoc.ents. Here we remove overlapping spans usingspacy.util.filter_spans. When spans overlap, the rule is to prefer the first longest span over shorter ones.