[Bug]: Local tiktoken fails writing to non-writeable directory in sandboxed macOS application

[tnunamak]

What happened?

litellm writes to the filesystem during initialization. I'm building a macOS application and this causes a read-only filesystem error:

Failed to execute script 'main' due to unhandled exception: [Errno 30] Read-only file system: '/private/var/folders/n4/l_tnw4sn7s1572j1t3_vq6mh0000gp/T/AppTranslocation/B62E81E2-6E1E-4F1B-8595-DE7AB482C491/d/Selfie.app/Contents/Frameworks/litellm'

Some investigation shows that it's due to this code, (original PR here).

As a workaround, perhaps this behavior could be disabled or TIKTOKEN_CACHE_DIR could be left alone if it is already set?

Relevant log output

Failed to execute script '__main__' due to unhandled exception: [Errno 30] Read-only file system: '/private/var/folders/n4/l_tnw4sn7s1572j1t3_vq6mh0000gp/T/AppTranslocation/B62E81E2-6E1E-4F1B-8595-DE7AB482C491/d/Selfie.app/Contents/Frameworks/litellm'

Traceback:
Traceback (most recent call last):
  File "selfie/__main__.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "selfie/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "litellm/__init__.py", line 558, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller/loader/pyimo<…>

[maurobender]

I'm facing the same issue using the codium/pr-agent deployed to a AWS Lamda function (I've also opened an issue there: https://github.com/Codium-ai/pr-agent/issues/909). Since the lamda filesystem is read-only (except for the /tmp folder) I'm getting this error. I've tried the workaround of setting TIKTOKEN_CACHE_DIR to /tmp folder in the filesystem but because of the change tnunamak mentioned before is not possible because of these lines in the code that are overwriting that environment variable.

[Jonas1312]

I have pretty much the same issue with litellm 1.38.11 and tiktoken 0.7.0 in my docker container.

litellm patches the env var TIKTOKEN_CACHE_DIR, which causes tiktoken to crash:

File \"/opt/app-root/lib64/python3.11/site-packages/tiktoken/load.py\", line 74, in read_file_cached
    with open(tmp_filename, \"wb\") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/opt/app-root/lib64/python3.11/site-packages/litellm/llms/tokenizers/fb374d419588a4632f3f557e76b4b70aebbca790.e7ca2cd3-f9c8-438e-9f61-6e71d33173af.tmp'

[chai3]

watch #4119

[chai3]

@tnunamak @maurobender @Jonas1312

4119 released on v1.40.9

We can use environment variable CUSTOM_TIKTOKEN_CACHE_DIR

[fingon]

@tnunamak @maurobender @Jonas1312 #4119 released on v1.40.9 We can use environment variable CUSTOM_TIKTOKEN_CACHE_DIR

This comment seems off, it is just TIKTOKEN_CACHE_DIR but it doesn't override it?

[chai3]

@fingon Are you still having problems?

litellm/utils.py top level code

os.environ["TIKTOKEN_CACHE_DIR"] = os.getenv("CUSTOM_TIKTOKEN_CACHE_DIR", filename) 

https://github.com/BerriAI/litellm/pull/4119/files#diff-9661b0d8f1d9f48433f2323d1102e963d3098f667f7a25caea79f16fa282f6ddL52-R54

[fingon]

In our case, the beautiful fix was just to import litellm and set the TIKTOKEN_CACHE_DIR afterwards so this was mostly reference to looking at code (but I might have looked at earlier version).