danny-avila / LibreChat

Enhanced ChatGPT Clone: Features Anthropic, AWS, OpenAI, Assistants API, Azure, Groq, o1, GPT-4o, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, langchain, DALL-E-3, ChatGPT Plugins, OpenAI Functions, Secure Multi-User System, Presets, completely open-source for self-hosting. Actively in public development.
https://librechat.ai/
MIT License
18.95k stars 3.15k forks source link

Enhancement: Upload Documents as input Context vs RAG Workflow #2755

Open achhabra2 opened 5 months ago

achhabra2 commented 5 months ago

What features would you like to see added?

It would be great if we could upload PDFs or text documents and have it processed as input context versus the current workflow which uses the RAG API instead. Some models now have 200K - 1M context window and we could utilize that outside of pasting large blocks of text into the chat window.

More details

We may need to have a secondary upload button implemented or something that signifies which type of workflow you are using. If you use chatGPT or Gemini web interface, etc. those documents get processed as context.

Below added from #3791


In the Claude UI, if you paste a large piece of text, it automatically gets attached and treated like a document. This is very easy to use as I can just paste large pieces of text from different sources, they get treated as separate documents, and then chat with them.

But LibreChat is more like ChatGPT, where any amount of pasted text gets added in the text box like a normal message. So having the above behavior I think is beneficial in many ways, at least as a toggle switch in the settings.

Sorry if this is duplicated; I couldn't find anything like this in the Issues. Loving LibreChat so far; really great alternative to paying for ChatGPT, Claude, and Gemini separately. Thanks!

Paste a large amount of text (threshold could be customizable maybe) and it will get uploaded as a TXT file instead of appearing in the chatbox.

Second, when clicking on such a file, a UI popup opens up where we can check the file.

Which components are impacted by your request?

No response

Pictures

image

This is what I'm referring to.

Code of Conduct

danny-avila commented 5 months ago

This request already exists, but your specifications are more clear so I will close the other in favor of yours. Thanks for the write up!

Closing https://github.com/danny-avila/LibreChat/issues/2335

raphaelgurtner commented 5 months ago

+1, this would be a major improvement, especially for use with gemini 1.5 models with their large context sizes. If it helps: supported mime types for each model can be found here https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/send-multimodal-prompts#media_requirements

danny-avila commented 5 months ago

Google will get a lot of love soon due to their improved dev tools. May have something to do with @logankilpatrick joining the team 😊

wesselhuising commented 4 months ago

Hi, any update on this? I would like to send PDFs to the Gemini 1.5 pro model instead using the RAG API. Cheers.

marcelamsler commented 3 months ago

This would improve the value of LibreChat massively and we are looking forward to this. Is there any timeline you could share?

schnaker85 commented 3 months ago

upvote +1 🚀

amir-ghasemi commented 3 months ago

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

schnaker85 commented 3 months ago

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/prompts/createContextHandlers.js#L25 I think it still requires to use the RAG_API_URL to be setup.

amir-ghasemi commented 3 months ago

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/prompts/createContextHandlers.js#L25 I think it still requires to use the RAG_API_URL to be setup.

Yes, indeed. We still need the RAG for question answering against knowledgebases consisting of 1000s of documents so that feature should not go away. This flag and the existing endpoint in rag api allows for including the full document in context with minimal changes.

raphaelgurtner commented 2 months ago

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

A document included in the context in this way is however still subject to any preprocessing/text-extraction on LibreChat's part right? The idea of this feature request would be to circumvent that (if the user so desires) and have the model (endpoint) deal with the document as-is. This would allow many different use-cases / document types, not only pdf but even sound/video/csv etc. - basically whatever is supported by the model-endpoints.

Is that feature considered out of scope (since it's not in the roadmap currently at all) or just low priority? If it's just low priority, would help implementing it still be appreciated?

danny-avila commented 2 months ago

A document included in the context in this way is however still subject to any preprocessing/text-extraction on LibreChat's part right?

No I think it would be nice to have a simple "use full text" option while uploading. If it's text-based, the browser can handle it and the server never interacts with the file, other than adding it to the AI request, it would just get appended to the user message.

marcelamsler commented 2 months ago

Is there an update here? We would love to use Librechat to compare Contracts in PDF Format with Gemini 1.5.

o42o commented 1 month ago

+1

banjavi commented 1 month ago

+1

bsu3338 commented 1 month ago

Could this change incorporate the option to send images to RAG instead of having the model process it? Thinking of images with newspaper articles, document scans in image format. Maybe possible to do the OCR with RAG API and then include it in "use full text". I can put this in as a separate request. Like the option to choose to either use RAG or process content with model.

hksitorus commented 1 month ago

This would be really great! Some of our colleagues asked if they can summarize a doc/pdf. Which does not really working well with RAG workflow. This will be a really great use case for Gemini.

We may need to have a secondary upload button implemented or something that signifies which type of workflow you are using. If you use chatGPT or Gemini web interface, etc. those documents get processed as context.

I feel like secondary upload button (only for model that support it) is making more sense, so user can choose between RAG workflow or input context.

Xtrah commented 1 month ago

This feature would be amazing and is so far the only crucial limitation I'm having using LibreChat.

danny-avila commented 2 weeks ago

A native implementation for this is planned and I will work on it soon, in order to send text from files as part of the context.

schnaker85 commented 2 weeks ago

A native implementation for this is planned and I will work on it soon, in order to send text from files as part of the context.

the above PR for the closed issue(https://github.com/danny-avila/LibreChat/pull/4503) would allow to send the complete files as base64 encoded string in the requests for example for google models.

Is this what you mean or only text-files (eg. no PDF's)?

marcelamsler commented 1 week ago

Why was the Pull Request closed? It contains a working implementation, which could be used as a base. We really need this feature, as the usage is so limited without the possibility to upload Files.

danny-avila commented 1 week ago

Why was the Pull Request closed? It contains a working implementation, which could be used as a base. We really need this feature, as the usage is so limited without the possibility to upload Files.

the PR is open but it doesn't address this issue. The point is to pass text-based files into the prompt, not to pass it as base64