tiktoken does not work with blobfile==2.1.0

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.16k stars 751 forks source link

tiktoken does not work with blobfile==2.1.0 #209

Closed Praful932 closed 8 months ago

Praful932 commented 9 months ago

Produces errors when these commands are run

tiktoken.encoding_for_model("gpt-3.5-turbo") - ValueError: Unknown encoding cl100k_base
tiktoken.encoding_for_model("gpt-3.5-turbo-0613") - 'ABCMeta' object is not subscriptable

hauntsaninja commented 8 months ago

I can't reproduce / it's unlikely that changes in blobfile are causing "unknown encoding". Could you be more specific about what problem you're facing (e.g. provide stacktrace, Python version, etc)

hauntsaninja commented 8 months ago

blobfile 2.1.1 will fix the second error you're seeing (on Python 3.8). The "unknown encoding" seems potentially different, could you provide a repro?

Separately, I'll make it so that tiktoken uses blobfile even less than it already does.