token <-> index operation

Yoctol / strpipe

text preprocessing pipeline

Other

5 stars 0 forks source link

Closed SoluMilken closed 6 years ago

SoluMilken commented 6 years ago

token to index (with unk token)

init: vocab_size, vocabulary, unk_token, necessary_tokens
fit: (1) check unk token not in sentences (2) build limited size vocabulary from sentences + (add unk token + add important_tokens must have)
transform: map token to index based on vocabulary,
inverse_transform: index to token
state: token2index, index2token

stegben commented 6 years ago

Did you forget to add a op?

SoluMilken commented 6 years ago

added op token to index fa9235b