search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.06k stars 749 forks source link

<|endoftext|>，Why can't ChatGPT recognize it? #261

Closed Kayce001 closed 2 months ago

Kayce001 commented 4 months ago

image

hauntsaninja commented 2 months ago

Might be a result of a post-training intervention. Not really a question for this repo!