search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k stars 856 forks source link

SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url #281

Open sijiashen opened 7 months ago

sijiashen commented 7 months ago

Issue I ran encoding = tiktoken.get_encoding("cl100k_base") and encountered the following error: SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

Cause & Solution Our hypothesis is that we are not allowed to making a request to a host outside of the kubeflow VPC. Q1: Would you say this is the primary cause?

If the answer to Q1 is yes: Q2: Does tiktoken allow us to read from a file system artifact instead of making a web call?

polinabee commented 7 months ago

I have the same issue, did you find any resolution?

anithamudigoudar commented 7 months ago

I have the same issue, is there any update on this?

polinabee commented 7 months ago

I seem to be able to get rid of this issue by setting a more restrictive SSL token in openssl dir for my python

anithamudigoudar commented 7 months ago

@polinabee Can you please provide the steps to do it please

songwang41 commented 6 months ago

I also encountered this error? any solution?

nkilm commented 6 months ago

It's because of the network issue. You can manually download the .tiktoken file and specify the location of this file in tiktoken using TIKTOKEN_CACHE_DIR environment variable.

You can find a detailed guide here - https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer

wirapratamaz commented 3 months ago

It's because of the network issue. You can manually download the .tiktoken file and specify the location of this file in tiktoken using TIKTOKEN_CACHE_DIR environment variable.

You can find a detailed guide here - https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer

Hi man how can I fix the SSLError with self-signed certificates when trying to use tiktoken offline? Also, what's the best way to make sure the files it needs are cached properly?

nkilm commented 2 months ago

It's because of the network issue. You can manually download the .tiktoken file and specify the location of this file in tiktoken using TIKTOKEN_CACHE_DIR environment variable. You can find a detailed guide here - stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer

Hi man how can I fix the SSLError with self-signed certificates when trying to use tiktoken offline? Also, what's the best way to make sure the files it needs are cached properly?

You can download and keep the .tiktoken file in your project itself and set TIKTOKEN_CACHE_DIR to where the .tiktoken file resides.