search

Mintplex-Labs / anything-llm

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.
https://anythingllm.com
MIT License
22.99k stars 2.33k forks source link

[HELP] document-summarizer error message for some PDF files #1944

Open CedricGoby opened 2 months ago

CedricGoby commented 2 months ago

Hi,

I'm using the docker version of Anything LLM and Chroma

mintplexlabs/anythingllm:master
chromadb/chroma:0.5.1.dev173

The docker host is a small VM with 2 CPU, 4 Go RAM and no GPU.

The LLM is "mistral-large-2204", the vector database is "Chroma" and the embedder is the default "AnythingLLM Embedder".

I have 2 PDF files in my workspace : a small one and a bigger one.

If I ask the agent to summarize those PDF files it works for the small one :

[backend] info: [TELEMETRY SENT] {"event":"sent_chat","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"multiUserMode":false,"LLMSelection":"mistral","Embedder":"native","VectorDbSelection":"chroma","runtime":"docker"}}
[backend] info: [Event Logged] - sent_chat
[backend] info: [AgentHandler] Start 74e19caa-0964-40e5-bbf3-c7234e577ead::mistral:mistral-large-2402
[backend] info: [TELEMETRY SENT] {"event":"agent_chat_started","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"runtime":"docker"}}
[backend] info: [AgentHandler] Attached websocket plugin to Agent cluster
[backend] info: [AgentHandler] Attached chat-history plugin to Agent cluster
[backend] info: [AgentHandler] Attaching user and default agent to Agent cluster.
[backend] info: [AgentHandler] Attached rag-memory plugin to Agent cluster
[backend] info: [AgentHandler] Attached document-summarizer plugin to Agent cluster
[backend] info: [AgentHandler] Attached web-scraping plugin to Agent cluster
[backend] info: [AgentLLM - mistral-large-2402] Valid tool call found - running document-summarizer.
[backend] info: [AgentHandler] [debug]: @agent is attempting to call `document-summarizer` tool
[backend] info: [AgentLLM - mistral-large-2402] Function tool with exact arguments has already been called this stack.
[backend] info: [AgentLLM - mistral-large-2402] Will assume chat completion without tool call inputs.
[backend] info: [TELEMETRY SENT] {"event":"agent_chat_sent","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"runtime":"docker"}}
[backend] info: [AgentLLM - mistral-large-2402] Valid tool call found - running document-summarizer.
[backend] info: [AgentHandler] [debug]: @agent is attempting to call `document-summarizer` tool
[backend] info: [AgentLLM - mistral-large-2402] Function tool with exact arguments has already been called this stack.
[backend] info: [AgentLLM - mistral-large-2402] Will assume chat completion without tool call inputs.
[backend] info: [TELEMETRY SENT] {"event":"agent_chat_sent","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"runtime":"docker"}}

But there is an error for the bigger PDF file :

[backend] info: [TELEMETRY SENT] {"event":"sent_chat","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"multiUserMode":false,"LLMSelection":"mistral","Embedder":"native","VectorDbSelection":"chroma","runtime":"docker"}}
[backend] info: [Event Logged] - sent_chat
[backend] info: [AgentHandler] Start 4cb58d33-0893-4826-9571-573677d9446d::mistral:mistral-large-2402
[backend] info: [TELEMETRY SENT] {"event":"agent_chat_started","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"runtime":"docker"}}
[backend] info: [AgentHandler] Attached websocket plugin to Agent cluster
[backend] info: [AgentHandler] Attached chat-history plugin to Agent cluster
[backend] info: [AgentHandler] Attaching user and default agent to Agent cluster.
[backend] info: [AgentHandler] Attached rag-memory plugin to Agent cluster
[backend] info: [AgentHandler] Attached document-summarizer plugin to Agent cluster
[backend] info: [AgentHandler] Attached web-scraping plugin to Agent cluster
[backend] info: [AgentLLM - mistral-large-2402] Valid tool call found - running document-summarizer.
[backend] info: [AgentHandler] [debug]: @agent is attempting to call `document-summarizer` tool
Failed to calculate number of tokens, falling back to approximate count Error: Unknown model
    at getEncodingNameForModel (/app/server/node_modules/js-tiktoken/dist/lite.cjs:230:13)
    at encodingForModel (/app/server/node_modules/@langchain/core/dist/utils/tiktoken.cjs:23:59)
    at ChatOpenAI.getNumTokens (/app/server/node_modules/@langchain/core/dist/language_models/base.cjs:182:75)
    at LLMChain._getNumTokens (/app/server/node_modules/langchain/dist/chains/llm_chain.cjs:212:44)
    at MapReduceDocumentsChain._call (/app/server/node_modules/langchain/dist/chains/combine_docs_chain.cjs:180:73)
    at async MapReduceDocumentsChain.invoke (/app/server/node_modules/langchain/dist/chains/base.cjs:58:28)
    at async summarizeContent (/app/server/utils/agents/aibitat/utils/summarize.js:55:15)
    at async Object.summarizeDoc (/app/server/utils/agents/aibitat/plugins/summarize.js:157:22)
    at async Object.handler (/app/server/utils/agents/aibitat/plugins/summarize.js:63:22)
    at async AIbitat.handleExecution (/app/server/utils/agents/aibitat/index.js:630:22)
[backend] error: (node:112) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
[backend] info: [AgentHandler] document-summarizer.summarizeDoc raised an error. 422 status code (no body)
[backend] info: [AgentLLM - mistral-large-2402] Function tool with exact arguments has already been called this stack.
[backend] info: [AgentLLM - mistral-large-2402] Will assume chat completion without tool call inputs.
[backend] info: [TELEMETRY SENT] {"event":"agent_chat_sent","distinctId":"0ecf90ec-3ef0-4b37-97a6-bc15ff5d91b4","properties":{"runtime":"docker"}}

Can you help me to understand what's wrong in that case ?

Regards

timothycarambat commented 2 months ago

🤔 Hm, this is weird. Looks like the looping or failing to do the recursive summarization due to the model context being overflow but then being unable to even estimate tokens.

[backend] error: (node:112) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit

Is actually normal and is coming from langchain - so the real issue is why LC cannot do the fallback calculation on tokens.

timothycarambat commented 2 months ago

Looks like this was patched in later version of @langchain/community or @langchain/core so we need to bump those deps to patch this