fix: panic with cohere tokenizer - Githubissues

daulet / tokenizers

Go bindings for HuggingFace Tokenizer

MIT License

92 stars 23 forks source link

fix: panic with cohere tokenizer #7

Closed jpekmez closed 1 year ago

jpekmez commented 1 year ago

exact tokenizers package version
fix tokenizer memory leak
fix panic in encode/decode with invalid utf8 string
fix empty string with special tokens test