Simplify Conversation History Management and Switch to Redis Checkpointers

[Teneroy]

Description

The goal of this task is to simplify the conversation history management system by removing Redis LangChain and using Redis Checkpointers instead. This will bring simplicity to the way we manage conversation state. Additionally, we will enable the companion to handle context changes (e.g., tab switches).

Tasks

1. Remove Redis LangChain History:

   • Eliminate the use of Redis LangChain History from all parts of the system

2. Integrate Redis Checkpointers:

   • Implement Redis Checkpointers as the sole method of managing conversation history and state checkpoints.
   • Ensure the integration is complete across all endpoints.

3. Remove Conversation Operations from Initialization Endpoint:

   • Update the initialization endpoint to no longer perform operations related to Redis history storage.
   • Ensure that the conversation state is correctly initialized without redundant history being stored at this point.

4. Filter Conversation History for the Planner:

   • Modify the planner to use a streamlined version of the conversation history, including only user inputs and companion outputs.

5. Add Dynamic Context Injection:

   • Implement functionality to dynamically inject context into the conversation history based on relevant changes (e.g., new resources being accessed or switched).
   • Ensure that injected context is necessary and does not clutter the history, leading to token overload.

6. Handle Context Updates for Tab Switching(example: you are looking at the pod, and then switched to API rules):

   • Add logic to detect when a user switches tabs or views (or changes resources) and update the companion’s context accordingly.
   • Ensure that relevant resource information is provided without storing unnecessary data into the conversation history.

Acceptance Criteria

• Remove Redis LangChain history operations and replace them with Redis Checkpointers.
• Remove conversation operations with Redis from the initialization endpoint.
• Filter conversation history to include only user input and companion output for the planner, removing intermediate or redundant messages.
• Implement dynamic context injection into the conversation history when necessary, ensuring that we do not overload the token usage with redundant or outdated context.
• Ensure that when users switch tabs or views, the companion is updated with relevant information, such as changes to resources, without polluting the conversation history with excessive data.
• Ensure that conversation history remains lightweight(less than 15k tokens) while still providing necessary context for accurate companion responses.

[Teneroy]

Don't forget to update tests after doing your changes :)