As far as I am able to tell, tiktoken-rs is able to encode into a vector of token references, and then you can decode those token references back into your original text. However my use case is to split my text into decoded tokens. I don't see a way to do this via the api. Having a method like "split_by_tokens" would be super helpful.
As far as I am able to tell, tiktoken-rs is able to encode into a vector of token references, and then you can decode those token references back into your original text. However my use case is to split my text into decoded tokens. I don't see a way to do this via the api. Having a method like "split_by_tokens" would be super helpful.