search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
649 stars 49 forks source link

Error importing in React Chrome Extension: Field 'browser' doesn't contain a valid alias configuration #14

Closed arcticfly closed 1 year ago

arcticfly commented 1 year ago

Thanks for making this extension!

I'm currently trying to import the @dqbd/tiktoken npm package in a React-based Chrome extension, and I'm running into the following error when I try to load my extension:

Module not found: Error: Can't resolve '@dqbd/tiktoken' in 'browser-extension/src/pages/Popup'
resolve '@dqbd/tiktoken' in 'browser-extension/src/pages/Popup'
  Parsed request is a module
  using description file: browser-extension/package.json (relative path: ./src/pages/Popup)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      browser-extension/src/pages/Popup/node_modules doesn't exist or is not a directory
      browser-extension/src/pages/node_modules doesn't exist or is not a directory
      browser-extension/src/node_modules doesn't exist or is not a directory
      looking for modules in browser-extension/node_modules
        single file module
          using description file: browser-extension/package.json (relative path: ./node_modules/@dqbd/tiktoken)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken is not a file
            .jpg
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.jpg doesn't exist
            .jpeg
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.jpeg doesn't exist
            .png
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.png doesn't exist
            .gif
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.gif doesn't exist
            .eot
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.eot doesn't exist
            .otf
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.otf doesn't exist
            .svg
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.svg doesn't exist
            .ttf
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.ttf doesn't exist
            .woff
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.woff doesn't exist
            .woff2
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.woff2 doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.js doesn't exist
            .jsx
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.jsx doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.ts doesn't exist
            .tsx
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.tsx doesn't exist
            .css
              Field 'browser' doesn't contain a valid alias configuration
              browser-extension/node_modules/@dqbd/tiktoken.css doesn't exist
        existing directory browser-extension/node_modules/@dqbd/tiktoken
          using description file: browser-extension/node_modules/@dqbd/tiktoken/package.json (relative path: .)
            using exports field: ./dist/bundler/_tiktoken.mjs
              using description file: browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/package.json (relative path: ./_tiktoken.mjs)
                no extension
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs doesn't exist
                .jpg
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.jpg doesn't exist
                .jpeg
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.jpeg doesn't exist
                .png
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.png doesn't exist
                .gif
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.gif doesn't exist
                .eot
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.eot doesn't exist
                .otf
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.otf doesn't exist
                .svg
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.svg doesn't exist
                .ttf
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.ttf doesn't exist
                .woff
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.woff doesn't exist
                .woff2
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.woff2 doesn't exist
                .js
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.js doesn't exist
                .jsx
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.jsx doesn't exist
                .ts
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.ts doesn't exist
                .tsx
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.tsx doesn't exist
                .css
                  Field 'browser' doesn't contain a valid alias configuration
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs.css doesn't exist
                as directory
                  browser-extension/node_modules/@dqbd/tiktoken/dist/bundler/_tiktoken.mjs doesn't exist
      node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
 @ ./src/pages/Popup/Popup.tsx 10:0-22 27:184-186
 @ ./src/pages/Popup/index.jsx 6:0-28 8:21-26

Has anyone seen this before? It seems that the npm package may not be getting compiled correctly, since the .mjs files referenced in package.json's export block don't seem to exist:

"exports": {
    ".": {
      "node": {
        "types": "./dist/node/_tiktoken.d.ts",
        "default": "./dist/node/_tiktoken.js"
      },
      "default": {
        "types": "./dist/bundler/_tiktoken.d.js",
        "default": "./dist/bundler/_tiktoken.mjs"
      }
    },
    "./bundler": {
      "types": "./dist/bundler/_tiktoken.d.ts",
      "default": "./dist/bundler/_tiktoken.mjs"
    },
    "./web": {
      "types": "./dist/web/_tiktoken.d.ts",
      "default": "./dist/web/_tiktoken.js"
    },
    "./node": {
      "types": "./dist/node/_tiktoken.d.ts",
      "default": "./dist/node/_tiktoken.js"
    }
  },
dqbd commented 1 year ago

Hello @arcticfly 😄 , sorry for the typo in the browser part, decided to scrap it entirely, replacing it with exports. 

npm i @dqbd/tiktoken@^1.0.0
arcticfly commented 1 year ago

Thanks for getting back to me. When I run this with version 1.0.0, I get the following error:

ERROR in ./node_modules/@dqbd/tiktoken/tiktoken_bg.wasm 1:0
Module parse failed: Unexpected character '' (1:0)
The module seem to be a WebAssembly module, but module is not flagged as WebAssembly module for webpack.
BREAKING CHANGE: Since webpack 5 WebAssembly is not enabled by default and flagged as experimental feature.
You need to enable one of the WebAssembly experiments via 'experiments.asyncWebAssembly: true' (based on async modules) or 'experiments.syncWebAssembly: true' (like webpack 4, deprecated).
For files that transpile to WebAssembly, make sure to set the module type in the 'module.rules' section of the config (e. g. 'type: "webassembly/async"').
(Source code omitted for this binary file)
 @ ./node_modules/@dqbd/tiktoken/tiktoken.js 1:0-43 3:15-19
 @ ./src/pages/Popup/TextToJS.tsx 16:0-52 18:12-30
 @ ./src/pages/Popup/Popup.tsx 12:0-34 28:212-220
 @ ./src/pages/Popup/index.jsx 10:0-28 14:26-31

webpack 5.75.0 compiled with 1 error in 7602 ms
dqbd commented 1 year ago

It seems like you need to add asyncWebAssembly in your Webpack configuration, see: https://github.com/dqbd/tiktoken/tree/main/js#nextjs for NextJS for instance.

Are you using a (meta)framework of sorts? Create React App or something similar?

arcticfly commented 1 year ago

I'm building a chrome extension that's based on Create React App. This error did go away after I added

experiments: {
  asyncWebAssembly: true,
},

To my webpack.config.js file.

Unfortunately, I then faced an issue specific to Chrome extensions with content security:

Uncaught (in promise) CompileError: WebAssembly.instantiateStreaming(): Refused to compile or instantiate WebAssembly module because neither 'wasm-eval' nor 'unsafe-eval' is an allowed source of script in the following Content Security Policy directive: "script-src 'self'"

I resolved this by editing the content security policy in my extension's manifest.json according to this post: https://groups.google.com/a/chromium.org/g/chromium-extensions/c/sJiaTnFMLHQ/m/y-qT1gplHwAJ

Thank you for helping me through this! I'll probably be back later with more questions.

dqbd commented 1 year ago

Glad it helped! Closing this issue for now.