Open weepy opened 1 year ago
Can you provide more context? The project itself is definitely more like a starting off point for people who want to use the ChatGPT api so there is definitely room for improvement, but I myself wasn't getting any errors with where it was at.
https://github.com/botisan-ai/gpt3-tokenizer
here is the repo for the tokenizer
running this - I get the following stack trace when i submit something .
TypeError: text.match is not a function at GPT3NodeTokenizer.encode (/Users/jonahfox/Downloads/chatgpt-svelte-main/node_modules/.pnpm/gpt3-tokenizer@1.1.5/node_modules/gpt3-tokenizer/dist/gpt3-tokenizer.cjs.development.js:155:26) at Module.getTokens (/src/lib/utils/tokenizer.ts:8:28) at eval (/src/routes/api/chat/+server.ts:28:44) at Array.forEach ()