Replaced the hardcoded URL https://openaipublic.blob.core.windows.net with the TIKTOKEN_BPE_HOST environment variable, allowing for flexibility in sourcing BPE data.
This change is particularly beneficial for environments where external access is restricted, such as private VPCs, or where organizations prefer using private/internal artifact repositories to pull dependencies. With this update, users can specify their own host URL for BPE data via TIKTOKEN_BPE_HOST, ensuring compatibility with network policies and internal infrastructure.
Additionally, it helps resolve SSL certificate verification errors like:
requests.exceptions.SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
By allowing the BPE host URL to be set internally, this change supports environments using private artifact repositories, ensuring seamless access to required files without SSL-related interruptions.
Replaced the hardcoded URL
https://openaipublic.blob.core.windows.netwith theTIKTOKEN_BPE_HOSTenvironment variable, allowing for flexibility in sourcing BPE data.This change is particularly beneficial for environments where external access is restricted, such as private VPCs, or where organizations prefer using private/internal artifact repositories to pull dependencies. With this update, users can specify their own host URL for BPE data via
TIKTOKEN_BPE_HOST, ensuring compatibility with network policies and internal infrastructure.Additionally, it helps resolve SSL certificate verification errors like:
By allowing the BPE host URL to be set internally, this change supports environments using private artifact repositories, ensuring seamless access to required files without SSL-related interruptions.