issues
search
Yoctol
/
strpipe
text preprocessing pipeline
Other
5
stars
0
forks
source link
added zh char tokenizer op
#33
Closed
SoluMilken
closed
6 years ago
SoluMilken
commented
6 years ago
tokenizer op
init
: input words to be tokenized and get tokenizer by id
fit
: pass
transform
: tokenizer.lcut
inverse_transform
: list of string to string (join).
state
: stateless.
tokenizer op
init: input words to be tokenized and get tokenizer by idfit: passtransform: tokenizer.lcutinverse_transform: list of string to string (join).state: stateless.