Closed Tymek closed 2 years ago
Ahh, thanks for submitting the issue. Currently it is built with NodeJS only packages and I need to swap out implementation for the TextEncoder and TextDecoder in order for browser usage. I will spend some time this week to do it.
I think there is an npm package called lib0 that can resolve the issue. If you have some extra time please give a hand and take a crack at it! Cheers!
@Tymek Please try the latest v1.1.2, it should work on the browser now. Please let me know if there are any more issues!
Hello @lhr0909 I have tested your update https://github.com/Niputi/vite-tests/tree/npm-gpt3-tokenizer
.mjs
import GPT3Tokenizer from 'gpt3-tokenizer'
as you use default export.util
, so why import from util
?fyi. if you want to export more stuff later, it's better export code as named exports instead of defaults as default and named exports dont go well together
This is what worked for me (node v18.1.0
):
import GPT3Tokenizer from 'gpt3-tokenizer'
const tokenizer = new GPT3Tokenizer.default({ type: 'gpt3' })
const str = "hello 👋 world 🌍";
const encoded = tokenizer.encode(str);
const decoded = tokenizer.decode(encoded.bpe);
console.log(encoded);
console.log(decoded);
No matching export in "browser-external:util" for import "TextEncoder"
in Vite (SvelteKit).Edit: this might be related: https://github.com/vitejs/vite/pull/6493