Allow option to load in encodings ref from local file path instead of external call

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.61k stars 785 forks source link

Allow option to load in encodings ref from local file path instead of external call #178

Closed nanditamohan closed 8 months ago

nanditamohan commented 1 year ago

This is to enable private network scenarios in AzureML RAG, where external public network calls (e.g. to https://openaipublic.blob.core.windows.net) will be restricted. In this case, we would refer tiktoken to use the encodings file from the image itself.

nanditamohan commented 1 year ago

@hauntsaninja could you please help review these changes and help us with the process to release this in tiktoken? Thank you!

nanditamohan commented 1 year ago

@hauntsaninja ping on this PR - please help review as we have dependency on this change for an upcoming deadline.

Mansoorkhango commented 1 year ago

HOW ARE YOU

hauntsaninja commented 8 months ago

Closing, since it's been a few months without a response