[FEAT]: Expand summarization to generic LLM Provider

FBTCAD commented 5 months ago

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

using LM Studio Community/Meta-Llama-3-8B-Instruct-GGUF/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf in LM Studio when trying to summarize a pdf file asked for API key. [2024-05-10 19:57:37.216] [INFO] [LM Studio Community/Meta-Llama-3-8B-Instruct-GGUF/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf] Accumulated 108 tokens: I apologize for the inconvenience, but I was unable to summarize the document MBThesis.pdf due to an error.\n\nError:\nAn error occurred while attempting to process the file. The necessary API keys (OpenAI or Azure OpenAI) were not found, which prevented me from accessing and analyzing the document.\n\nIf you would like to provide me with a valid API key or an alternative method to summarize the document, I'll be happy to assist you further. Alternatively, please let me know if there's anything else I can help you

Here is the output from AnythingLLM: @agent summary of MBThesis.pdf

Agent @agent invoked. Swapping over to agent chat. Type /exit to exit agent execution loop early. [debug]: @agent is attempting to call document-summarizer tool @agent: Looking at the available documents. @agent: Found 1 documents @agent: Grabbing all content for MBThesis.pdf @agent: Summarizing MBThesis.pdf... I apologize for the inconvenience, but I was unable to summarize the document MBThesis.pdf due to an error.

Error: An error occurred while attempting to process the file. The necessary API keys (OpenAI or Azure OpenAI) were not found, which prevented me from accessing and analyzing the document.

If you would like to provide me with a valid API key or an alternative method to summarize the document, I’ll be happy to assist you further. Alternatively, please let me know if there’s anything else I can help you with.

Are there known steps to reproduce?

No response

timothycarambat commented 5 months ago

I think this is a limitation with the doc-summarize tool. We need to make it more generic to be able to just use the current client. Will address this today

eriksonssilva commented 4 months ago

@timothycarambat I was facing the same issue. Then I decided to start AnythingLLM from terminal (I'm on Linux). Before that, I made sure to use 'export OPENAI_API_KEY="dummy-key"'. Once I did that, the agent always gave me error 401 complaining about the API Key not being valid: or

Note that in my case I am using Ollama and trying to mimic what you have explained here: https://www.youtube.com/watch?v=4UFrVvy7VlA

I appologize if I should not be bumping this topic since you told you're already working on it. I was just making sure to give you some more context.

Thanks a lot!

Propheticus commented 4 months ago

Found the same and reported here https://discord.com/channels/1114740394715004990/1243182212170645544

timothycarambat commented 4 months ago

https://github.com/Mintplex-Labs/anything-llm/blob/acbad2f2cf7c13e03e0a0dc1067041ba4c59930c/server/utils/agents/aibitat/utils/summarize.js#L13

Propheticus commented 4 months ago

https://github.com/Mintplex-Labs/anything-llm/blob/acbad2f2cf7c13e03e0a0dc1067041ba4c59930c/server/utils/agents/aibitat/utils/summarize.js#L13

Well that's as clear cut an explanation as they come 😄 No words needed.

estcap2 commented 4 months ago

I am getting the same response as @FBTCAD . I tried latest native macOS and Docker versions, different PDFs, diferent models etc.

Example using Ollama and llama3:8b-instruct-q8_0:

@agent can you please summarize DavydovTayar.pdf?
Agent @agent invoked. Swapping over to agent chat. Type /exit to exit agent execution loop early.
[debug]: @agent is attempting to call `document-summarizer` tool
@agent: Looking at the available documents.
@agent: Found 1 documents
@agent: Grabbing all content for DavydovTayar.pdf
@agent: Summarizing DavydovTayar.pdf...
I apologize, but it seems that I encountered an issue while attempting to summarize the file “DavydovTayar.pdf”. Unfortunately, I was unable to access the necessary information due to a missing OpenAI or Azure OpenAI API key.

It also asks for OpenAI key, the weird part is that the same document can be accessed via traditional Rag in the same chat without the agent, but I am looking to replicate the results seen here https://www.youtube.com/watch?v=4UFrVvy7VlA

Update: Trying to run the same with a disabled internet connection, it throws a timeout:

@timothycarambat could it be that the tool is not 100% run locally?

estcap2 commented 4 months ago

I am getting the same response as @FBTCAD . I tried latest native macOS and Docker versions, different PDFs, diferent models etc.

Example using Ollama and llama3:8b-instruct-q8_0:
@agent can you please summarize DavydovTayar.pdf?
Agent @agent invoked. Swapping over to agent chat. Type /exit to exit agent execution loop early.
[debug]: @agent is attempting to call `document-summarizer` tool
@agent: Looking at the available documents.
@agent: Found 1 documents
@agent: Grabbing all content for DavydovTayar.pdf
@agent: Summarizing DavydovTayar.pdf...
I apologize, but it seems that I encountered an issue while attempting to summarize the file “DavydovTayar.pdf”. Unfortunately, I was unable to access the necessary information due to a missing OpenAI or Azure OpenAI API key.
It also asks for OpenAI key, the weird part is that the same document can be accessed via traditional Rag in the same chat without the agent, but I am looking to replicate the results seen here https://www.youtube.com/watch?v=4UFrVvy7VlA

Update: Trying to run the same with a disabled internet connection, it throws a timeout:

@timothycarambat could it be that the tool is not 100% run locally?

🤦‍♂️ nevermind I just took a look to the tool's code. For anyone wondering, at the moment the summarization is not run locally, it uses OpenAI's GPT. My confusion arose from the cited video and the notion I had that everything was run locally

Mintplex-Labs / anything-llm