search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k stars 856 forks source link

tiktoken does not work with blobfile==2.1.0 #209

Closed Praful932 closed 1 year ago

Praful932 commented 1 year ago

Produces errors when these commands are run

tiktoken.encoding_for_model("gpt-3.5-turbo") - ValueError: Unknown encoding cl100k_base
tiktoken.encoding_for_model("gpt-3.5-turbo-0613") - 'ABCMeta' object is not subscriptable
hauntsaninja commented 1 year ago

I can't reproduce / it's unlikely that changes in blobfile are causing "unknown encoding". Could you be more specific about what problem you're facing (e.g. provide stacktrace, Python version, etc)

hauntsaninja commented 1 year ago

blobfile 2.1.1 will fix the second error you're seeing (on Python 3.8). The "unknown encoding" seems potentially different, could you provide a repro?

Separately, I'll make it so that tiktoken uses blobfile even less than it already does.