Streamlined package - Githubissues

cldf / segments

Unicode Standard tokenization routines and orthography profile segmentation

Apache License 2.0

31 stars 13 forks source link

Closed xrotwang closed 7 years ago

xrotwang commented 7 years ago

This PR

refactors Tokenizer
- such that Profile and Rules become separate objects, which makes initialization more modular,
- adds some configurability of input and output,
- unifies tokenize and tokenize_ipa into a single method, implemented in __call__ (because tokenizing seems to be the default thing to do for a Tokenizer).
adds a cli, with two sub-commands tokenize and profile

closes #1

xrotwang commented 7 years ago

Thanks for merging - hope you are not frustrated by my ever-changing views of what API is best :)

bambooforest commented 7 years ago

Not at all -- sorry for the slow merge (was returning to Z). Mattis is here now, so it was a good time to look it over and discuss next topics.