Python memory usage

[logan-markewich]

Am I measuring this properly? Is tiktoken using 500MB of memory?

If so, is there any way to control this?

[GaurangTandon]

Hi @logan-markewich did you find any workaround for this?

The increased memory usage tends to crash small Docker containers, so this is really a bottleneck in adopting tiktoken. Is there an alternative that consumes 10x less memory, even if it has slightly reduced accuracy? (in token counting)

[hauntsaninja]

I can't reproduce:

λ cat x.py
import psutil
process = psutil.Process()

def get_memory():
    print(process.memory_info().rss / 1000000)

get_memory()
import tiktoken
get_memory()
tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo").encode
get_memory()
λ python x.py
15.097856
18.997248
78.41792

I tried on a couple different systems. All were roughly about^

Could you provide more information about your system configuration? (Also please don't post screenshots, it makes reproducing your code harder)

[GaurangTandon]

Thanks @hauntsaninja you're right. My comment was based on an older version of tiktoken, and a different setting than a Python script. I will try to reproduce the issue again and post here (perhaps in a separate issue)

[logan-markewich]

@hauntsaninja sorry, I should have mentioned the screenshot above was running in colab. Let me try again on a few other machines

[logan-markewich]

@hauntsaninja here is a link to the colab, this reproduces every time I run it: https://colab.research.google.com/drive/1JnEHblVIPz534yRDcJFD-lTWspXyUKpy?usp=sharing

I also ran locally (macOS, M2 Pro Max, python3.11) and got a similar output

159.744
193.00352
914.06336 <--- !!!

[logan-markewich]

@hauntsaninja wow, I was missing a zero 😢 Turns out it's not a huge deal then tbh. Closing out for now