Open obo opened 3 years ago
It is a good idea. There are two approaches for dealing with different tokenizer idea. First approach: we can make various tokenizers in the SLTev and identified them with numbers or names. Second approach: we can allow users to use a file that contains a function for tokenization.
Anyone who would like
SLTev
to support custom tokenizers (e.g. via--tokenizer=...
), please discuss here. Let's add only features people need. Pull requests are also welcome.