Closed fsa317 closed 4 months ago
Let me take a look.
Looks like I got past this by directly doing an npm install on gcc and make.
On Wed, Jun 12, 2024 at 12:22 PM Adhityan K V @.***> wrote:
Let me take a look.
— Reply to this email directly, view it on GitHub https://github.com/llm-tools/embedJs/issues/76#issuecomment-2163744535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7CHDBHSJDZUEPLYZ47TZ3ZHCNYRAVCNFSM6AAAAABJG6AF4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTG42DINJTGU . You are receiving this because you authored the thread.Message ID: @.***>
This looks like an issue with one of the dependencies. They have a PR addressing this but it has not been merged for months (https://github.com/mscdex/mmmagic/pull/171). I will try and get the upstream patched; if not, I will publish a version using alternative package or a forked version.
The install of mmmagic works for me but then when I run the app I created I get:
node:internal/modules/cjs/loader:1072
const err = new Error(message);
^
Error: Cannot find module '../build/Release/magic'
Require stack:
- /Users/apapf/MyCode/RagTest/node_modules/mmmagic/lib/index.js
- /Users/apapf/MyCode/RagTest/node_modules/stream-mmmagic/lib/stream-mmmagic.js
-
I notice there is no Release directory inside node_modules/mmmagic/build directory. Is that expected?
Published a new version (0.0.89
) with a patched version of mmagic
. NPM Install is working fine in my mac and tested the library in linux. Could you confirm if issue is resolved for you?
Published a new version (
0.0.89
) with a patched version ofmmagic
. NPM Install is working fine in my mac and tested the library in linux. Could you confirm if issue is resolved for you?
So the install worked, but still having issues using things. I am trying to install this in a nextjs app by the way.
So now the npm install works. Then I did npm install hnswlib-node
. However when I try to import { HNSWDb } from '@llm-tools/embedjs/vectorDb/hnswlib-db';
it does not work, basically says it can't find it. I see the file inside a similar directory in node_modules but with dist
in the path.
What's interesting is that if I change the import to import { HNSWDb } from '@llm-tools/embedjs/vectorDb/hnswlib';
(remove the -db) I get an error in my IDE but then at runtime I get a different error:
Module not found: Can't resolve '../build/Release/magic'
https://nextjs.org/docs/messages/module-not-found
Import trace for requested module:
./node_modules/stream-mmmagic-patched/lib/stream-mmmagic.js
Yes, the import path is import { HNSWDb } from '@llm-tools/embedjs/vectorDb/hnswlib'
. You can refer to all the import paths in the documentation. This is done so that people need to install only the minimum dependencies they need. The end goal here is to move to a monorepo setup.
Actually, NextJs has some issues with node native packages. I am currently actually testing this out and building a sample app here. It's still a WIP but I hope to get support added soon.
Thanks for the update. I think I’ll check back when this works for nextjs.
On Fri, Jun 14, 2024 at 7:09 AM Adhityan K V @.***> wrote:
Yes, the import path is import { HNSWDb } from @.***/embedjs/vectorDb/hnswlib'. You can refer to all the import paths in the documentation https://github.com/llm-tools/embedJs?tab=readme-ov-file#hnswlib.
Actually, NextJs has some issues with node native packages. I am currently actually testing this out and building a sample app here https://github.com/llm-tools/chat-bot-nextjs-template. It's still a WIP but I hope to get support added soon.
— Reply to this email directly, view it on GitHub https://github.com/llm-tools/embedJs/issues/76#issuecomment-2167796201, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7CHDCMFTRKGVELQLTSJ3DZHLFP5AVCNFSM6AAAAABJG6AF4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRXG44TMMRQGE . You are receiving this because you authored the thread.Message ID: @.***>
Will post an update when NextJs is supported. Closing the issue for now.
So finally have this working with NextJs. Check the example here with pinecone - https://github.com/llm-tools/chat-bot-nextjs-template.
Point to note - not all vector databases work with NextJs. It may be possible to get each of them wotking with specific config changes to the bundler but by default some of them will not work. The base library itself will work fine out of the box. Beyond that, several API based vector databases work out of the box.
Trying to install the package on my mac and get the following error: