Closed jannisborn closed 1 year ago
Add support for bigsmiles tokenization based on a regexp. Favoring simple regexp compared to bigsmiles.tokenizer.text_tokenize in bigsmiles repo for speed reasons
bigsmiles.tokenizer.text_tokenize
Add support for bigsmiles tokenization based on a regexp. Favoring simple regexp compared to
bigsmiles.tokenizer.text_tokenizein bigsmiles repo for speed reasons