tokenizer.ts doesn't need a conditional import if you're using Vite (npm run dev), but it is necessary when using (npm run build --debug && npm run preview).
So, why does tokenizer.ts require this conditional check to import GPT3TokenizerImport?
However – the following Just Works™️ when you're in npm run dev in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:
// $lib/tokenizer.ts
import GPT3Tokenizer from 'gpt3-tokenizer';
const tokenizer = new GPT3Tokenizer({ type: 'gpt3' });
export function getTokens(input: string): number {
const tokens = tokenizer.encode(input);
return tokens.text.length;
}
So, why does tokenizer.ts require this conditional check to import
GPT3TokenizerImport
?The code below (from the GitHub issue mentioned in the video), checks if you're using
CommonJS
or not.However – the following Just Works™️ when you're in
npm run dev
in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:I hope this helps.