Some local LLMs are very good at summarization tasks. The one currently in RAGged Edge Box (Stability LM) was chosen for its question extraction performance.
If the local LLM is good enough for summarization, it would be interesting to add chunk summarization.
It could be added either at document ingestion or on-demand when showing the search results.
Having direct access to the LLM (#11) will help experimentation but it is not necessary.
Similar to the RAG prompt, the summarization prompt ought to be configurable (#14).
Some local LLMs are very good at summarization tasks. The one currently in RAGged Edge Box (Stability LM) was chosen for its question extraction performance.
If the local LLM is good enough for summarization, it would be interesting to add chunk summarization.
It could be added either at document ingestion or on-demand when showing the search results.
Having direct access to the LLM (#11) will help experimentation but it is not necessary.
Similar to the RAG prompt, the summarization prompt ought to be configurable (#14).