wasm version or pure python version needed

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

12.27k stars 830 forks source link

wasm version or pure python version needed #134

Closed mccoysc closed 1 year ago

mccoysc commented 1 year ago

in pyodide(a python runtime running in web page or nodejs),only wasm or pure python version supported. to run llama_index in this python version,wasm or pure python code needed.

mccoysc commented 1 year ago

and for "lib.rs",it's easy to be compiled into wasm. rust can support wasm target perfectly.

hauntsaninja commented 1 year ago

See https://github.com/openai/tiktoken/issues/94#issuecomment-1498263172