search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
648 stars 49 forks source link

js-tiktoken ES5 support #64

Closed haolong7 closed 11 months ago

haolong7 commented 11 months ago

Hey there. My project is using ES5 as compile target for tsconfig. Which would complain an error for https://github.com/dqbd/tiktoken/blob/c86963f5222caf32dcd7b50d3137a8053d710c61/js/src/core.ts#L141 Just wondering is there a way to provide ES5 support? If necessary, I can provide a PR for this.

dqbd commented 11 months ago

Hello @haolong7!

No plans to support ES5 unfortunately. The reason being that the .cjs output file will include unnecessary polyfill code, which results in more code being parsed regardless if ES6 features are available or not.

Would it be possible to instead configure the bundler to transpile dependencies as well? Feel free to comment if I'm missing something or further guidance is needed 🙂.

haolong7 commented 11 months ago

Hi @dqbd.

Thanks for the explanation. Yeah, move the App to support ES6 is the right way to go. Meanwhile, I found a workaround by using older version of js-token(1.0.2) which supports ES5.