ceifa / tiktoken-node

OpenAI's tiktoken but with node bindings
103 stars 10 forks source link

UnhandledSchemeError when building module in Next.js #13

Closed BlockchainFreak closed 1 year ago

BlockchainFreak commented 1 year ago

Description: I encountered an UnhandledSchemeError when attempting to build a module in my Next.js project. The error message reads:

"Module build failed: UnhandledSchemeError: Reading from "node:module" is not handled by plugins (Unhandled scheme). Webpack supports "data:" and "file:" URIs by default. You may need an additional plugin to handle "node:" URIs."

I am using Windows as my operating system, Mozilla Firefox as my browser, and have React 18.2.0 and Next.js 13.3.0 installed.

import { encodingForModel } from "tiktoken-node"
const calculateTokens = async (messages: { role: "user" | "assistant", content: string }[]) => {

    let numTokens = 0
    // causes error
    const gptEncoder = encoding_for_model("gpt-3.5-turbo")
    ...
    return numTokens
}

Steps to Reproduce:

  1. Create a Next.js project with React 18.2.0
  2. Attempt to build a module that uses "node:" URI
  3. Encounter UnhandledSchemeError

Expected Behavior: The module should build successfully without encountering any errors.

Actual Behavior: Encountered an UnhandledSchemeError and the module failed to build.

Environment:

ceifa commented 1 year ago

This is not a tiktoken-node issue, please track it on https://github.com/vercel/next.js/issues/28774