Custom tokenizer as a separate module

ybracke / transnormer

A lexical normalizer for historical spelling variants using a transformer architecture.

GNU General Public License v3.0

6 stars 1 forks source link

Closed ybracke closed 1 year ago

ybracke commented 1 year ago

[x] Move customized normalizer for huggingface tokenizers ("transliterator") from train_model.py into its own module (--> transnormer.preprocess.translit)
[x] Create tests
[x] Use new functionality in train_model.py