Option to use env variable for BPE_FILE location

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.76k stars 801 forks source link

Option to use env variable for BPE_FILE location #128

Open csnelsonchu opened 1 year ago

csnelsonchu commented 1 year ago

Currently the BPE_FILE is hardcoded to the the path https://openaipublic.blob.core.windows.net/.... This host path is presents challenge when running in container in AWS VPC environment. It will be great if we can just specify the location of the file using ENV variable so we avoid such issues.