[Question] Difference between `encode_with_special_tokens` and `encode_ordinary`

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

MIT License

240 stars 46 forks source link

Open kojix2 opened 1 year ago

kojix2 commented 1 year ago

Hi @zurawiki

I have a question today.

Could you tell me the difference between the two functions? They both seem to allow special tokens.