search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
705 stars 53 forks source link

How do I pack this into a JS that I can use directly in the browser? #39

Closed ghost closed 1 year ago

ghost commented 1 year ago

like that

<script src="tiktoken.js"></script>
<script>
const enc = Tiktoken.get_encoding("gpt2");
</script>
ghost commented 1 year ago

tried https://esm.sh/tiktoken 

caught TypeError: B.__wbindgen_add_to_stack_pointer is not a function
    at Module.L (tiktoken.bundle.mjs:1:3977853)
    at <anonymous>:1:7
dqbd commented 1 year ago

Hello @InterestingDarkness!

The setup for a browser is most likely determined by the way how you bundle the app / whether you bundle at all. For simplicity sake, it might be a better idea to use the accompanying js-tiktoken package, which does not include any WASM binaries at all.

ghost commented 1 year ago

It works. Thank you!

import * as Tiktoken from 'https://cdn.skypack.dev/js-tiktoken?min'
const enc = Tiktoken.encodingForModel("gpt-4");