Netlify deploy fails for ssr things it don't need

[CNSeniorious000]

It always ends like this.

[vite-plugin-pwa:build] Cannot bundle Node.js built-in path imported from node_modules/.pnpm/tiktoken@1.0.10/node_modules/tiktoken/lite/tiktoken.cjs. Consider disabling ssr.noExternal or remove the built-in dependency.

Through tests, I think this must be removed otherwise Netlify will still try to ssr that module and fail:

export const initTikToken = tiktoken_bg_wasm
  ? async() => {
    const { init } = await import('tiktoken/lite/init')
    const [{ bpe_ranks, special_tokens, pat_str }, { Tiktoken }] = await Promise.all([
      getBPE(),
      import('tiktoken/lite/init'),
      fetch(tiktoken_bg_wasm).then(r => r.arrayBuffer()).then(wasm => init(imports => WebAssembly.instantiate(wasm, imports))),
    ])
    return new Tiktoken(bpe_ranks, special_tokens, pat_str)
  }
  : async() => {
    const [{ bpe_ranks, special_tokens, pat_str }, { Tiktoken }] = await Promise.all([
      getBPE(),
-     import('tiktoken/lite'), // currently this will cause ssr error on netlify
    ])
    return new Tiktoken(bpe_ranks, special_tokens, pat_str)
  }

Currently I haven't figure out better ways to avoid this annoying bug.

[CNSeniorious000]

Since 556766d, tiktoken_bg.wasm and cl100k_base.json will be loaded from node_modules or CDN instead or using import.

So this bug is finally solved. 🎉