openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k stars 856 forks source link

Nuitka compiler, unknown encoding "ck100_base" #207

Closed aspoofer0224 closed 1 year ago

aspoofer0224 commented 1 year ago

enabled_plugin option:[] is empty and couldnt load the encoding model and so far i couldnt find a solution to it, i've tried importing tiktoken_ext as a package in command but it didnt work like pyinstaller case. Highly appreciate if someone had succeed in building tiktoken with nuitka.

hauntsaninja commented 1 year ago

See the FAQ for pyinstaller, I assume you'll need to do something similar https://github.com/openai/tiktoken/issues/98