search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

BPE Memory leak #199

Closed venual closed 1 year ago

venual commented 1 year ago

I dont know if im using it wrong but when creating a new BPE it creates around 20MB of memory and never releases it, on top of that the async_openai::get_max_tokens_chat_message function creates a new bpe in it so big memory usage that never releases after every call

venual commented 1 year ago

wrong repo my bad, this was meant for tiktoken-rs