search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.61k stars 784 forks source link

BPE Memory leak #199

Closed venual closed 11 months ago

venual commented 11 months ago

I dont know if im using it wrong but when creating a new BPE it creates around 20MB of memory and never releases it, on top of that the async_openai::get_max_tokens_chat_message function creates a new bpe in it so big memory usage that never releases after every call

venual commented 11 months ago

wrong repo my bad, this was meant for tiktoken-rs