Make encode token generic, accepting usize and u32

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

MIT License

240 stars 47 forks source link

Open mokeyish opened 10 months ago

mokeyish commented 10 months ago

Candle does not support usize, and it is quite troublesome to do type conversion every time.

zurawiki commented 10 months ago

Thanks for the feedback. Makes sense to support a generic type here. I don't have bandwidth this weekend but PRs are welcome