Integrate into MONAPipe - Githubissues

ybracke / transnormer

A lexical normalizer for historical spelling variants using a transformer architecture.

GNU General Public License v3.0

6 stars 1 forks source link

Integrate into MONAPipe #73

Open ybracke opened 8 months ago

ybracke commented 8 months ago

Write a spacy component that makes use of the normalization method for MONAPipe. Look here for the introduction to writing such a component.

First idea:

Apply the normalizer to sentences (doc.sents), if necessary split sentences first into <512-byte-chunks.
Apply an aligner to normalized and original sentence
- Apply some function to resolve 1:n, m:1 and m:n mappings
Add the aligned words as attributes of tokens (Token._.)