Add support for summarization

[taranjeet]

• Since the data is splitted in chunks, is it possible to implement a summarize function?
• opened on behalf of discord user edo, message link

[edops1]

Hi! I'll try to explain the context where I would use the tool. For work I write a lot of interviews (for TV or podcasts). I would like to upload a varied documents about a character (interviews from youtube, books, articles). Then I would ask for these things:

• the summary of all the documents together
• the summary of a single document (specifying a compression ratio to the length of the original document). Sometimes I would like a summary of a few sentences, other times I would like to have a 50% reduction so as not to miss important passages

With a set of summaries, I am clear on the big picture and can start asking specific questions to sharpen certain passages in that character's life

[edops1]

Hi, I was reading the documentation of langchain and found this document:

Vector store-augmented text generation https://python.langchain.com/docs/modules/chains/additional/vector_db_text_generation

Is this something that can also be implemented in embedchain?

Since documents (pdf, web pages etc) are already divided into chunks and added to the vector database for q&a, could it be simple to generate new text from these? @taranjeet

Thanks!

[sv-ochis]

any updates on this ?