add encode_batch api and make all encode api support (char) offsets

daulet / tokenizers

Go bindings for HuggingFace Tokenizer

MIT License

85 stars 23 forks source link

Closed sunhailin-Leo closed 10 months ago

sunhailin-Leo commented 1 year ago

Add encode_batch and Update encode API.
- return_offsets: will return token position.
- with_char_mode: when return_offsets is True, it will return char position.