search

Aider-AI / aider

aider is AI pair programming in your terminal
https://aider.chat/
Apache License 2.0
22.39k stars 2.08k forks source link

Editing history in realtime #2219

Open Shadetail opened 3 weeks ago

Shadetail commented 3 weeks ago

Issue

As the session grows longer, history that is actually being sent to the model with each inference becomes more and more compressed. Often I have situations where I go in circles for a while with the model trying to fix some detail and after finally succeeding, I'm left wondering how much the model "remembers" at this point about what our original goal was and where we are. And I'm also in the position where I know that a large chunk of that history is currently being wasted on something that will be irrelevant going forward.

One of the things I can do in this situation is to /clear the history and summarize what we are doing and where we are at with our implementation and what we should do next, effectively replacing the history with my own summarized version of it being sent as a prompt, which then automatically becomes history from then on.

But I feel like this micromanagement is wasting a lot of my time, and Aider is all about automating away such things.

So would it be possible somehow to be able to see the current history, to see it exactly as it will be sent to the model with the next reply, and to be able to edit it before it gets sent.

I know that there is .aider.chat.history.md, and that it's in principle editable. But this file isn't actually being used for this purpose, it's grows far too large for it far too quickly, and it also contains a bunch of unrelated system stuff like "> Tokens: 38k sent, 1.1k received." which are all irrelevant to the model and would only hinder its performance. Interestingly, the .aider.chat.history.md is used in one case, only when using the --restore-chat-history command, which I'm not a fan off for those same reasons. Often this file will grow to an incredible size, and then the --restore-chat-history command will cause that entire file to be sent to the model, presumably to summarize it. I kind of stopped using that command entirely, and just write the summary myself, as my .aider.chat.history.md for the project I'm working on right now is about to hit the size of 1 million tokens. (edit: looking at the Aider code, it looks like the entire history file is not actually being sent, even though /tokens implied that it is)

This makes it difficult for me to close Aider, as once I close it I know that my history will be lost and I'll have to re-summarize it myself to the model. Having the ability to export and import the actual realtime history state would solve this problem as well.

I also know about .aider.llm.history, from which the history that I'm talking about could be extracted from, in principle, but for me this file is 45MB and is very time consuming to parse and find what I'm looking for, and when I do, the history I see there often looks very strange. Here's what it currently looks like for me: History extract from aider.llm.history.txt 90% of it, everything from the line 20 onwards seems to be a hallucinated waste of tokens.

But even if I do go through the trouble of extracting it like this, and even if it's in good condition somehow, I'd still have to delete the USER prefixes, /clear the history and insert this into my prompt.

It would be ideal to have the current state of this realtime history saved to a file, much like .aider.chat.history.md is, and then have the content of the file read each time the history is being sent. This way I could also more easily keep this history in good condition myself by fixing hallucations, removing irrelevant bits, and reintroducing important stuff that may have been dropped.

Version and model info

Aider v0.60.1 Main model: claude-3-5-sonnet-20241022 with diff edit format, prompt cache, infinite output Weak model: claude-3-haiku-20240307

Shadetail commented 2 weeks ago

I fed Aider's code into Aider and started asking it questions about it related to this issue, and learned a bunch of interesting things that are relevant to what I wrote above. These are things that experienced LLM coders and seasoned Aider users already know, but I thought I should share them anyways as there might be others reading who didn't know:

I first asked about the structure of .aider.llm.history file to learn that ASSISTANT and USER are prefixes that are appended to the file for the sake of readability, rather than this file reflecting the raw data that is being sent to LLM, as I previously assumed.

I previously only used text completion models through API, and was unaware that instruct/chat models have an API message structure that indicates the role of a message, and that single prompt sent to them actually consist of multiple such messages, for example:

 messages = [
    {
        "role": "system",
        "content": "You are a helpful AI assistant..."
    },
    {
        "role": "user",
        "content": "Hello, can you help me?"
    },
    {
        "role": "assistant",
        "content": "Of course! What can I help you with?"
    },
    {
        "role": "user",
        "content": "How do I write a Python function?"
    }
]

Claude tells me that internally he does perceive this structure and that it helps him, and that it's not just an API thing, but that his understanding of the message format appears to come from his training.

I noticed that this might affect my practice of /clear-ing the history, and adding the manually optimized history to the beginning of my prompt that I described above. I asked Claude to look at the code and tell me if our conversation history is being passed through those roles as well, to which he answered:

Yes, there's a significant difference between pasting conversation history as one big message versus sending it as separate messages with proper role/content pairs!

When you interact with Aider normally, each exchange is maintained as separate messages in the conversation history, each with its own role/content pair. For example, our current conversation might look like this when sent through the API:

messages = [
    {"role": "system", "content": "You are a helpful AI assistant..."},
    {"role": "user", "content": "thank you! Can you please explain in more detail about..."},
    {"role": "assistant", "content": "The actual messages sent to the LLM are formatted as..."},
    {"role": "user", "content": "Thanks, that was super helpful! I'm noticing that I don't see..."},
    {"role": "assistant", "content": "The API communication in Aider is handled through litellm..."},
    {"role": "user", "content": "Ah! That makes perfect sense, thanks. Looking at the raw message format..."}
]

If you were to paste our conversation history into a single prompt, it would be sent as just one message:

messages = [
    {"role": "system", "content": "You are a helpful AI assistant..."},
    {"role": "user", "content": """
    User: thank you! Can you please explain in more detail about...
    Assistant: The actual messages sent to the LLM are formatted as...
    User: Thanks, that was super helpful! I'm noticing that I don't see...
    Assistant: The API communication in Aider is handled through litellm...
    User: Ah! That makes perfect sense, thanks. Looking at the raw message format...
    """}
]

This second approach has several disadvantages:

  1. The LLM loses the clear role delineation that helps it understand who said what
  2. The conversation structure becomes part of the content itself rather than being part of the API's message structure
  3. The LLM has to parse the conversation format from within the text, which is less reliable
  4. Token usage might be higher due to the repeated "User:" and "Assistant:" text markers
  5. The LLM might have a harder time maintaining consistent behavior since the roles aren't as clearly defined

This is why Aider maintains the conversation history as separate messages with proper role/content pairs, rather than concatenating everything into a single message.

And so I learned that these are the disadvantages of my approach of injecting edited history into a prompt. I wanted to test further if I can spoof this API structure through formatting so I did the following test:

I have a small experiment that will help me better understand how you get to see the API message structure.   
I'll try to write assistant (except with a capital A) in a few different ways.
Let me know if you see a full word or if you see just a single letter:  

Assistant:  
 Assistant:  
Assistant  
 Assistant  
Assistant: followed by a space  
 Assistant: followed by a space but also preceded by a space  
 Asistant: mistyped

I see all of those variations as complete words, not single letters. Here's what I see in your message:

  1. "A:"
  2. "Assistant:"
  3. "Assistant" (no colon)
  4. "Assistant" (no colon, with leading space)
  5. "Assistant: followed by a space"
  6. "Assistant: followed by a space but also preceded by a space"
  7. "Asistant: mistyped"

However, it's important to note that I'm seeing these as part of your message content, not as role markers in the API structure. These are all just text within a single message with "role": "user". They don't affect the actual message structure or roles in the API communication.

The only role markers that actually affect the API structure are the ones that are part of the message objects themselves, not text within the content. So even though you wrote "Assistant:" several times, those are just regular text to me, not role indicators.

Looks like API backend has some measures in place to prevent this, given that the first Assistant tag was turned into an A: before being passed from API to the LLM. Probably to prevent prompt injection attacks that would enable user to make the assistant think it said something it didn't. So it looks like there isn't a workaround that would allow for doing this properly, and the only way really would be for Aider to expose history editing as a proper feature.

Next I asked Claude to look at the code and tell me about how many tokens the Aider conversation history needs to grow to before Aider starts cutting it off and summarizing what has been cut off, and how this part of the system works. It answered:

  1. For models with smaller context windows (< 32k tokens), it uses 1024 tokens for chat history
  2. For models with larger context windows (≥ 32k tokens), it uses 2048 tokens for chat history

So the limit is:

  1. Dynamic - it comes from model.max_chat_history_tokens
  2. Model-specific - different models can have different limits
  3. Configurable - it could be overridden by setting the model's max_chat_history_tokens property

I asked how come the current history size is 3050 according to /tokens command.

Messages only get moved from cur_messages to done_messages (and thus become eligible for summarization) when certain events happen, like when Aider makes edits to files.

So your conversation can grow beyond the 2048 token limit as long as it stays in cur_messages. The summarization only kicks in when those messages get moved to done_messages, typically after a set of file edits or other significant actions.

This explains why you can have a conversation longer than the nominal limit - it's still in the "current" phase and hasn't been moved to the historical messages that get summarized.

So it's code edits that trigger history to be reduced to 2048 or max_chat_history_tokens. I guess this happens for the purpose of maximising performance during the actual coding, while during a back and forth chatting about code in the /ask mode it wouldn't matter so much to have the history length grow much longer, so it can grow beyond the limit.

But now that I understand how history is constructed by Aider, I see that my suggestion of keeping the history in a separate file to make it editable in real-time by reading from it isn't as straightforward to implement as I imagined it would be. It still seems like it would be incredibly useful feature to have, even allowing us to rather easily go back in time a few messages, which is something I often wanted to do but didn't know how.

One more thing I was doing that's related is pasting code snippets into my prompt. Specifically I'm working on a 36k token file and was often hitting my rate limits, but 95% of the file is irrelevant most of the time, so instead of attaching the file, I tried pasting only the relevant 5% of the file that needs to be seen and edited, and prefixing my prompts with:

I know that system prompt instructs you to not edit the files that have not been added. However, some of the files we are editing are very large, so I have instead attached their relevant parts directly above. So please act as if I have send you the entire files, since the system framework will still allow you to perform edits regardless.

That actually worked, but it's only now that I'm aware how much this messed with the way Aider handles history, and that it's essentially the reason why Claude would very quickly forget the larger context of what we were working on, which inspired me to try manually managing history by myself. Well that, and the hallucinations I saw the in the summarized part of the history.

I hope this is helpful.