chroma-core / chroma

the AI-native open-source embedding database
https://www.trychroma.com/
Apache License 2.0
15.66k stars 1.32k forks source link

Nextjs invalid exports #824

Closed jeffchuber closed 1 year ago

jeffchuber commented 1 year ago

From discord: https://discord.com/channels/1073293645303795742/1130258136905814066/1130258136905814066

./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js
Module not found: Package path . is not exported from package C:\personal-dev\ai-prototypes-2\node_modules@visheratin\web-ai (see exports field in C:\personal-dev\ai-prototypes-2\node_modules@visheratin\web-ai\package.json)

Import trace for requested module:
./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js

Anyone know how i can make webpack or nextjs not through these module not found issues? It seems like it shouldn't be importing this unless the function is actually being ran?
dosu-bot commented 1 year ago

Answer generated by a 🤖

Answer

From your description, it appears that you're encountering an issue with Next.js where it's unable to find the module @visheratin/web-ai. This is likely due to the package not correctly exporting its main file, which is a common problem when using ES6 imports with CommonJS modules.

To address this, you could modify the package.json file of the @visheratin/web-ai package to include an exports field. This field should specify the main file to be exported when the package is imported. Here's an example of how you could set it up:

{
  "name": "@visheratin/web-ai",
  "version": "1.0.0",
  "main": "index.js",
  "exports": {
    ".": "./index.js"
  }
}

In this example, the exports field is indicating that the main file (index.js) should be exported when the package is imported.

However, if you don't have the ability to modify the package.json of the @visheratin/web-ai package, you could try using a dynamic import() statement instead of a static require() statement. This can sometimes bypass issues with ES6 imports and CommonJS modules.

Here's an example of how you could adjust the WebAIEmbeddingFunction class to use a dynamic import() statement:

export class WebAIEmbeddingFunction implements IEmbeddingFunction {
  // ...
  constructor(
    modality: "text" | "image",
    node: boolean,
    proxy?: boolean,
    wasmPath?: string,
    modelID?: string
  ) {
    if (node) {
      this.proxy = proxy ? proxy : false;
      import("@visheratin/web-ai-node")
        .then((module) => {
          webAI = module;
        })
        .catch((e) => {
          console.log(e);
          throw new Error(
            "Please install the @visheratin/web-ai-node package to use the WebAIEmbeddingFunction, `npm install -S @visheratin/web-ai-node`"
          );
        });
    } else {
      this.proxy = proxy ? proxy : true;
      import("@visheratin/web-ai")
        .then((module) => {
          webAI = module;
        })
        .catch((e) => {
          console.log(e);
          throw new Error(
            "Please install the @visheratin/web-ai package to use the WebAIEmbeddingFunction, `npm install -S @visheratin/web-ai`"
          );
        });
    }
    // ...
  }
  // ...
}

In this revised version of the WebAIEmbeddingFunction class, the require() statements have been replaced with import() statements. These import() statements return a promise that resolves to the imported module, which is then assigned to the webAI variable.

Please note that this is just one potential solution to your problem. If you find this solution to be correct and believe it could benefit other users, I encourage you to make a pull request. If this solution doesn't align with your experience, I recommend updating to the latest version of the software.

Sources


This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

e-roy commented 1 year ago

I got this error with using Next JS using the newer app router

I rolled back to "chromadb": "1.5.2", and no longer saw the error.

cishiv commented 1 year ago

I got this error with using Next JS using the newer app router

I rolled back to "chromadb": "1.5.2", and no longer saw the error.

Chiming in here - confirmed that rolling back to 1.5.2 has worked around the error on my end.

jeffchuber commented 1 year ago

@cishiv thanks!! if anyone can help me with a small reproducible example - id love to fix this

e-roy commented 1 year ago

@cishiv thanks!! if anyone can help me with a small reproducible example - id love to fix this

Hi @jeffchuber

Here is a repo: https://github.com/e-roy/langchain-chatbot-demo

This is currently using "chromadb": "1.5.2"

If I upgrade to 1.5.5, it will still work, but there will be Module not found errors in the terminal

chroma-error

If I go through the process of adding each dependency @visheratin/web-ai-node, @visheratin/web-ai and encoding, then I'll get the error from the original post.

I hope this helps

jeffchuber commented 1 year ago

thank you @e-roy ! asking Vercel for help on this

shuding commented 1 year ago

From a quick look I think the problem is that the CJS build of chromadb was used (dist/main here instead of dist/module):

./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js

And then, a require is called:

require("@visheratin/web-ai")

So it tries to load the CJS build of @visheratin/web-ai. However, in that package's exports field, it has only types and import conditions for the . path defined: https://unpkg.com/browse/@visheratin/web-ai@1.2.3/package.json

"exports": {
  ".": {
    "types": "./index.d.ts",
    "import": "./index.js"
  },

Which doesn't have a CJS build provided for that path at all. To fix, you have to either use import() to load the ESM build, or add a "main" field to its package.json, or add a "require" condition (https://nodejs.org/api/packages.html#conditional-exports).

visheratin commented 1 year ago

@shuding, thank you for the lead! I just opened a PR that addresses the issue with require/import statements. I tested the fixed version, and it works in both browser and Node.js.

perzeuss commented 1 year ago

From a quick look I think the problem is that the CJS build of chromadb was used (dist/main here instead of dist/module):

./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js

And then, a require is called:

require("@visheratin/web-ai")

So it tries to load the CJS build of @visheratin/web-ai. However, in that package's exports field, it has only types and import conditions for the . path defined: https://unpkg.com/browse/@visheratin/web-ai@1.2.3/package.json

"exports": {
  ".": {
    "types": "./index.d.ts",
    "import": "./index.js"
  },

Which doesn't have a CJS build provided for that path at all. To fix, you have to either use import() to load the ESM build, or add a "main" field to its package.json, or add a "require" condition (https://nodejs.org/api/packages.html#conditional-exports).

You will still have this error when you use the ESM build. See

- wait compiling /_error (client and server)...
- error ../chroma/clients/js/dist/module/embeddings/WebAIEmbeddingFunction.js:132:0
Module not found: Can't resolve '@visheratin/web-ai'
  130 |         try {
  131 |             // @ts-ignore
> 132 |             const webAI = await import("@visheratin/web-ai");
  133 |             if (wasmPath) {
  134 |                 webAI.SessionParams.wasmRoot = wasmPath;
  135 |             }

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
../chroma/clients/js/dist/module/index.js
./src/app/page.js

The problem here is, that the package @visheratin/web-ai is not installed which is intended because this module is optional and there is a try-catch around that import to catch that error at runtime. However, the Next.js build script tries to locate the module during the compile process and throws this error because it is not installed.

@shuding can you confirm that?

visheratin commented 1 year ago

I just checked and can confirm that this issue exists. When working on a fix, I didn't consider this use case.

To fix the issue in version 1.5.6 of the package for Next.js, you can add the following to the Webpack config in next.config.js:

config.resolve.fallback = {
      "@visheratin/web-ai": false
    };

This way, the app will get compiled.

If somebody knows how to fix this issue in the WebAIEmbeddingFunction, I'd greatly appreciate the help.

perzeuss commented 1 year ago

I just checked and can confirm that this issue exists. When working on a fix, I didn't consider this use case.

To fix the issue in version 1.5.6 of the package for Next.js, you can add the following to the Webpack config in next.config.js:

config.resolve.fallback = {
      "@visheratin/web-ai": false
    };

This way, the app will get compiled.

If somebody knows how to fix this issue in the WebAIEmbeddingFunction, I'd greatly appreciate the help.

I already fixed the WebAIEmbeddingFunction in https://github.com/chroma-core/chroma/pull/956

GallagherSam commented 1 year ago

I was having this same issue on a Vue app I'm working on. I found the following script roughly fixes the import problem so my app will build properly. Not a great solution, but works.

input_file="node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js"
output_file="node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js"

while IFS= read -r line; do
    if [[ $line == *'require("@visheratin/'* ]]; then
        line="//${line}"
    fi
    echo "$line"
done < "$input_file" > "$output_file"
jeffchuber commented 1 year ago

we are close to fixing this up thanks to some excellent work by @perzeuss ! stand by :)

l4b4r4b4b4 commented 1 year ago

well unfortunately neither the PR was merged nor does the custom webpack config work for me. Would be great not to have to build a workaround or rolling back to prev. versions to fix the issue.

jeffchuber commented 1 year ago

This should be fixed in the release 1.5.8 - can folks give that a spin?

jonsoku-dev commented 1 year ago

@jeffchuber The error is continuously displayed in version 1.5.8 when ingest documents.

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

-  ┌ POST /api/chat/retrieval 200 in 21499ms
   │
   ├──── POST http://127.0.0.1:7080/api/v1/collections 200 in 32ms (cache: MISS)
   │
   ├── 1 level ── POST http://127.0.0.1:11434/api/generate 200 in 3215ms (cache: MISS)
   │
   └── 1 level ── POST http://127.0.0.1:7080/api/v1/collections/1038.. 200 in 66ms (cache: MISS)
perzeuss commented 1 year ago

@jeffchuber The error is continuously displayed in version 1.5.8 when ingest documents.

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

-  ┌ POST /api/chat/retrieval 200 in 21499ms
   │
   ├──── POST http://127.0.0.1:7080/api/v1/collections 200 in 32ms (cache: MISS)
   │
   ├── 1 level ── POST http://127.0.0.1:11434/api/generate 200 in 3215ms (cache: MISS)
   │
   └── 1 level ── POST http://127.0.0.1:7080/api/v1/collections/1038.. 200 in 66ms (cache: MISS)

@jonsoku2 Are you trying to use the webai embedding function or only the chroma client?

e-roy commented 1 year ago

This seems to be working fine now with Next JS, using: "chromadb": "^1.5.9", "langchain": "^0.0.150", "next": "^13.4.19",

Here is a working example:

https://github.com/e-roy/langchain-chatbot-demo/tree/main

For this error:

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

You can adjust your next.config.js file and ignore it by adding

  webpack: (config, {}) => {
    config.resolve.fallback = {
      "@visheratin/web-ai-node": false,
    };
    return config;
  },
l4b4r4b4b4 commented 1 year ago

This seems to be working fine now with Next JS, using: "chromadb": "^1.5.9", "langchain": "^0.0.150", "next": "^13.4.19",

Here is a working example:

https://github.com/e-roy/langchain-chatbot-demo/tree/main

For this error:

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

You can adjust your next.config.js file and ignore it by adding

  webpack: (config, {}) => {
    config.resolve.fallback = {
      "@visheratin/web-ai-node": false,
    };
    return config;
  },

interesting! Will try out. Thx for notifying!

jeffchuber commented 1 year ago

This is fixed with 1.6.1