feat: RAG hooks (e.g. upload_document)

[tjbck]

pipelines should have all the hooks required for enable custom end-to-end RAG for open-webui

[MarlNox]

Reposting for discussion purposes:

What would be the suggested approach to query the documents that OpenWebUI already has in memory? Currently, I'm "copying" the approach from the Apps/RAG section of the web UI, but it feels quite heavy (had to transplant lot of files/modules to pipelines to make it work), and fails to fetch collections by names, unless i put the hashed name in the search.

What would be a better way to conduct rag on the current documents we have on openwebui?

[tjbck]

That's a use case I haven't deeply thought about yet, but you should be able to use open webui API to do basic retrieval tasks. (check [open-webui url]/rag/api/v1/docs)

Could elaborate more on how you'd like to utilise pipelines and I'll make the appropriate changes if needed!

[MarlNox]

That's a use case I haven't deeply thought about yet, but you should be able to use open webui API to do basic retrieval tasks. (check [open-webui url]/rag/api/v1/docs)

Could elaborate more on how you'd like to utilise pipelines and I'll make the appropriate changes if needed!

Yep, Im currently working on making the pipeline i described in the enhancement. Basically the user asks a question, the system comes back with a response of sorts: "Here are a list of 5 documents which are the most relevant to your query: [list of docs]" The user chooses document 1 and 3 by communicating it to the agent (or maybe even through some sort of GUI interaction). Any further interactions with the agent will be search queries on the documents selected.

This would mean that the first query is used to run a search through the whole documents database. Then a few lines of communication will be used to steer the subsequent interactions based on user choice.