I want to modify the code in self._core_bpe.decode_bytes(tokens).decode("utf-8", errors=errors)

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.98k stars 816 forks source link

I want to modify the code in self._core_bpe.decode_bytes(tokens).decode("utf-8", errors=errors) #302

Closed FanshuoZeng closed 4 months ago

FanshuoZeng commented 4 months ago

If I want to modify the code in self._core_bpe.decode_bytes(tokens).decode("utf-8", errors=errors), then do I have to modify the _tiktoken.cpython-39-x86_64-linux-gnu.so file

hauntsaninja commented 4 months ago

Yes, methods on self._core_bpe are native code, not Python