Add subword segmentation support - Githubissues

Helsinki-NLP / OpusFilter

OpusFilter - Parallel corpus processing toolkit

MIT License

101 stars 18 forks source link

Add subword segmentation support #45

Closed svirpioj closed 2 years ago

svirpioj commented 2 years ago

BPE and Morfessor methods
support in preprocessing
support in language models