search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 47 forks source link

Allow users to supply their own BPE tokeniser #83

Open xd009642 opened 1 month ago

xd009642 commented 1 month ago

Following on from https://github.com/zurawiki/tiktoken-rs/issues/81#issuecomment-2288583643 in #81 I made our change part of the library API.