To save on token usage, we need to implement a method for scrubbing code snippets from the conversation history. The conversation history can grow large over time, particularly with LLM responses that include code. By identifying and removing old or unnecessary code snippets, we can reduce the token usage in subsequent LLM calls.
Observations from lib/aifunc.py and webwright/main.py
Relevant Code from lib/aifunc.py
This file encompasses functions to call AI models, handle function execution requests, and process chat completion requests from different APIs (OpenAI and Anthropic). It tracks and processes conversation history, which may include code responses from the LLMs.
Executing Functions: Code responses may include function calls.
Maintaining History: Each AI response along with the function results is appended to the messages list. This history could include blocks of code.
Relevant Code from webwright/main.py
This file establishes the main entry point of the application, maintains the main event loop, handles user prompts, and processes AI responses. The conversation history is built and maintained here, with each LLM response appended to the history, potentially growing large over time.
Building History: Maintains a conversation_history list containing user prompts and AI responses.
Displaying Responses: Prints formatted AI responses back to the user.
Tasks
[ ] Track code snippets within AI responses.
[ ] Periodically trim conversation history to remove old or unnecessary code snippets.
[ ] Optionally implement caching/referencing for relevant responses to maintain context.
Benefits
Reduced token usage by removing unnecessary code from conversation history.
Improved performance and cost efficiency for LLM calls.
Description
To save on token usage, we need to implement a method for scrubbing code snippets from the conversation history. The conversation history can grow large over time, particularly with LLM responses that include code. By identifying and removing old or unnecessary code snippets, we can reduce the token usage in subsequent LLM calls.
Observations from
lib/aifunc.pyandwebwright/main.pyRelevant Code from
lib/aifunc.pyThis file encompasses functions to call AI models, handle function execution requests, and process chat completion requests from different APIs (OpenAI and Anthropic). It tracks and processes conversation history, which may include code responses from the LLMs.
messageslist. This history could include blocks of code.Relevant Code from
webwright/main.pyThis file establishes the main entry point of the application, maintains the main event loop, handles user prompts, and processes AI responses. The conversation history is built and maintained here, with each LLM response appended to the history, potentially growing large over time.
conversation_historylist containing user prompts and AI responses.Tasks
Benefits