Pausing and resuming the summarization algorithm

[genaforvena]

One of the most pressing tasks for now is to enable pausing and resuming the summarization algorithm. This is because summarization takes a lot of time for large texts, and I have to run the algorithm from the start repeatedly, which is inefficient. The pause and resume feature would be very useful for this task. I also think it is important to think about knowledge trees and how we can store summaries better. Instead of plain text, we should use a format that allows us to search for summaries effectively.

Here are some elaborations and facts:

• Pausing and resuming the summarization algorithm: This is a feature that allows the user to stop the algorithm at any point and resume it later from where it left off. This can save time and resources, especially for large texts that take longer to summarize. One possible way to implement this feature is to use checkpoints, which are intermediate states of the algorithm that can be stored and loaded.
• Knowledge trees: These are hierarchical structures that organize information into nodes and branches. They can help to store summaries better by grouping them into categories and subcategories, and showing the relationships between them. For example, a knowledge tree of a book summary can have nodes for chapters, sections, characters, themes, etc. One advantage of using knowledge trees is that they can support query-based summarization, which allows the user to ask specific questions and get relevant summaries.

[genaforvena]

https://blog.reachsumit.com/posts/2023/09/generative-retrieval/

[genaforvena]

Naive implementation is done and working (at least haven't noticed fails)