Error: ENOENT: no such file or directory, open '/workspace/ChatGPT-Next-Web/.next/server/app/tiktoken_bg.wasm'

[Andision]

I am using tiktoken with Next.js as below like code metioned in #23

import { type TiktokenModel, get_encoding, encoding_for_model } from "@dqbd/tiktoken";

export const getChatGPTEncoding = (
  messages: Message[],
  model: string
) => {
  const isGpt3 = model === 'gpt-3.5-turbo' || model === 'gpt-3.5-turbo-0301';

  const encoder = encoding_for_model(model as TiktokenModel, {
    "<|im_start|>": 100264,
    "<|im_end|>": 100265,
    "<|im_sep|>": 100266,
  });

  const msgSep = isGpt3 ? '\n' : '';
  const roleSep = isGpt3 ? '\n' : '<|im_sep|>';

  const serialized = [
    messages
      .map(({ role, content }) => {
        return `<|im_start|>${role}${roleSep}${content}<|im_end|>`;
      })
      .join(msgSep),
    `<|im_start|>assistant${roleSep}`,
  ].join(msgSep);

  return encoder.encode(serialized, "all");
};

but I got error like this

error - Error: ENOENT: no such file or directory, open '/workspace/ChatGPT-Next-Web/.next/server/app/tiktoken_bg.wasm'
    at Object.openSync (node:fs:601:3)
    at Object.readFileSync (node:fs:469:35)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/utils.ts:17:72)
    at (sc_client)/./app/utils.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1171:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/store/app.ts:17:64)
    at (sc_client)/./app/store/app.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1127:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/store/index.ts:18:62)
    at (sc_client)/./app/store/index.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1138:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/components/home.tsx:20:65)
    at (sc_client)/./app/components/home.tsx (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:973:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
wait  - compiling...
event - compiled client and server successfully in 603 ms (1404 modules)
warn  - Fast Refresh had to perform a full reload due to a runtime error.
error - Error: ENOENT: no such file or directory, open '/workspace/ChatGPT-Next-Web/.next/server/app/tiktoken_bg.wasm'
    at Object.openSync (node:fs:601:3)
    at Object.readFileSync (node:fs:469:35)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/utils.ts:17:72)
    at (sc_client)/./app/utils.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1171:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/store/app.ts:17:64)
    at (sc_client)/./app/store/app.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1127:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/store/index.ts:18:62)
    at (sc_client)/./app/store/index.ts (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:1138:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)
    at eval (./app/components/home.tsx:20:65)
    at (sc_client)/./app/components/home.tsx (/workspace/ChatGPT-Next-Web/.next/server/app/page.js:973:1)
    at __webpack_require__ (/workspace/ChatGPT-Next-Web/.next/server/webpack-runtime.js:33:42)

I want to know how to fix this problem, thank you very much! Wish you a nice day!

[dqbd]

Hello @Andision! I think the issue here is the usage of experimental appDir, which is something not tested yet. Will look into it though.

Update: I believe there is a possible fix to introduce Next 13 App support, stay tuned for next release 😄

[arronKler]

try config your project like this

// next.config.js
const nextConfig = {
  reactStrictMode: true,
  webpack: function (config, options) {
        config.experiments = { asyncWebAssembly: true, syncWebAssembly: true };
        return config;
    },
}

module.exports = nextConfig

[natarius]

Facing the same issue when trying to deploy it to a Firebase Cloud Function. Is there any workarounds?

[natarius]

tried this in my webpack config but still getting same error

new LoaderOptionsPlugin({
    test: /[\/\\]node_modules[\/\\]@dqbd[\/\\]tiktoken[\/\\]tiktoken_bg\.wasm$/,
    options: {
        // via https://github.com/dqbd/tiktoken/issues/28
        asyncWebAssembly: true,
        syncWebAssembly: true,
    },
}),

[dqbd]

I am using tiktoken with Next.js as below like code metioned in #23

Released a prerelease version, try npm install @dqbd/tiktoken@1.0.5-preview, it should work with NextJS 13 with appDir without any further changes @Andision

Facing the same issue when trying to deploy it to a Firebase Cloud Function. Is there any workarounds?

Will look into it, haven't tested Firebase Cloud Functions yet, hopefully there will be a similar minor fix available. @natarius

[dqbd]

Just tried a quick test via firebase init functions and both firebase emulators:start --only functions and firebase deploy --only functions seem to work fine with @dqbd/tiktoken@^1.0.4

firebase/index.ts

import * as functions from "firebase-functions";
import { get_encoding } from "@dqbd/tiktoken";

export const helloWorld = functions.https.onRequest((request, response) => {
  const enc = get_encoding("gpt2");
  const result = new TextDecoder().decode(
    enc.decode(enc.encode("hello world"))
  );
  enc.free();
  functions.logger.info(result, { structuredData: true });
  response.send(result);
});

firebase.json

{
  "functions": [
    {
      "source": "functions",
      "codebase": "default",
      "runtime": "nodejs16",
      "ignore": [
        "node_modules",
        ".git",
        "firebase-debug.log",
        "firebase-debug.*.log"
      ],
      "predeploy": [
        "npm --prefix \"$RESOURCE_DIR\" run build"
      ]
    }
  ]
}

package.json

{
  "name": "functions",
  "scripts": {
    "build": "tsc",
    "build:watch": "tsc --watch",
    "serve": "npm run build && firebase emulators:start --only functions",
    "shell": "npm run build && firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "main": "lib/index.js",
  "dependencies": {
    "@dqbd/tiktoken": "^1.0.4",
    "firebase-admin": "^11.5.0",
    "firebase-functions": "^4.2.0"
  },
  "devDependencies": {
    "firebase-functions-test": "^3.0.0",
    "typescript": "^4.9.0"
  },
  "private": true
}

Would it be possible to share a reproducible repository? @natarius

[dqbd]

Support for Next 13 appDir has landed in 1.0.6. Closing this issue for now (as it has been solved for Next 13).

@natarius If the issue persist on Firebase Functions, feel free to comment here or create a new issue with a reproducible example.

[natarius]

What webpack version do you use?

we are on v4...might that be the issue?

[Andision]

Support for Next 13 appDir has landed in 1.0.6. Closing this issue for now (as it has been solved for Next 13).

@natarius If the issue persist on Firebase Functions, feel free to comment here or create a new issue with a reproducible example.

I update my dependencies to "@dqbd/tiktoken": "^1.0.6", in package.json and this problem is solved. Thank you so much!

But I got another error🤣, I think I am supposed to report it in a new issue.