SSL: Certificate_Verify_Failed

[Kwalladee]

I am trying to use TikToken on a work machine and TikToken I guess doesn't allow self-signed certificates. I believe they are like that due to my organization's policy but I'm wondering if there is a workaround for this other than to contact my organization's security team?

"requests.exceptions.SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)')))"

When trying to run this code:

import tiktoken as tk

model = "gpt-3.5-turbo-0301"
text = "How many tokens is this?"

encoding = tk.encoding_for_model(model)
print(len(encoding.encode(text)))```

[hauntsaninja]

tiktoken isn't doing anything special; it's literally just a requests.get: https://github.com/openai/tiktoken/blob/5d970c1100d3210b42497203d6b5c1e30cfda6cb/tiktoken/load.py#L24

If you can find another way to get the encoding files to the machine you need, you can put them in a local directory and set the TIKTOKEN_CACHE_DIR env var appropriately: https://github.com/openai/tiktoken/blob/5d970c1100d3210b42497203d6b5c1e30cfda6cb/tiktoken/load.py#L44

If that option doesn't work, I'd contact your security team