openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12k stars 818 forks source link

Nuitka compiler, unknown encoding "ck100_base" #207

Closed aspoofer0224 closed 11 months ago

aspoofer0224 commented 11 months ago

enabled_plugin option:[] is empty and couldnt load the encoding model and so far i couldnt find a solution to it, i've tried importing tiktoken_ext as a package in command but it didnt work like pyinstaller case. Highly appreciate if someone had succeed in building tiktoken with nuitka.

hauntsaninja commented 11 months ago

See the FAQ for pyinstaller, I assume you'll need to do something similar https://github.com/openai/tiktoken/issues/98