search

dqbd / tiktokenizer

Online playground for OpenAPI tokenizers
https://tiktokenizer.vercel.app
MIT License
717 stars 89 forks source link

Can't run the tiktokenizer from the app folder in next.js #14

Open gablabelle opened 1 year ago

gablabelle commented 1 year ago

Hello,

Trying to run tiktokenizer from the app folder and I'm getting the following error:

Error: Element type is invalid. Received a promise that resolves to: [object Promise]. Lazy element type must resolve to a class or function.

Here is the branch to reproduce the error.

Screenshot 2023-08-19 at 13 41 48

dqbd commented 1 year ago

Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of tiktoken.

One possible solution would be to replace tiktoken with js-tiktoken, if that's needed, but I'm still keen on experimenting with WASM and RSC 🙂

gablabelle commented 1 year ago

Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of tiktoken.

One possible solution would be to replace tiktoken with js-tiktoken, if that's needed, but I'm still keen on experimenting with WASM and RSC 🙂

I've since migrated to js-tiktoken and it works! 👍🏽