stevenlu137
commented
4 months ago Yes, I’m also working on improving AIlice’s ability to modify and edit code in complex software projects. It often overwrites previous work, and this problem is largely because AI lacks a suitable text editor. If there were a convenient text editor available, the LLM would likely prefer modify existing code than generating a new code file. Currently, there are a few solutions:
- Use BASH functions and text editing commands like sed.
- Provide a REPLACE function for text file editing.
- Parse the code into functions/classes and allow the LLM to replace the elements that need modification, which is the method you proposed.
The first method is natively supported but has the drawback that the LLM might need to use many escape characters in the text replacement expressions, which it is not very good at, leading to potential errors.
I have already implemented the second method but haven't yet submitted the corresponding changes in the prompt (so you can’t use it yet). This method works in tests but is not ideal yet.
The third method is more complex. Leveraging existing knowledge when providing tools for LLM is an effective technique. Teaching it to do complex things with prompts incurs significant costs and is less effective than using tools it has learned during pre-training. If we had a well-known command-line tool for structured code editing that the LLM is already familiar with, it would be an ideal choice—we would only need to give it a slight prompt. On the other hand, building a completely new tool and teaching it in the prompt could result in an overly complex prompt and potentially poor outcomes. However, if we could create a very simple tool, that would also be feasible.
This is still an ongoing experiment. I can upload the complete code to dev soon, and if you're interested, you can give it a try. I’m also interested in solutions for directly editing functions or classes in the code. Do we have a command-line tool for such code editing functions?
FellowTraveler
I noticed the agent was working on the code and the file got bigger, and bigger, say 3k, then 6k, then 9k. ...Then at some point the file was 2 or 3k again. Basically the agent just deleted everything it had done and rewrite the file. I think that code files should not be thought of as "files" to the agent. Instead, whether the agent writes the code, or ingests it, the agent should determine how to split up the file (into a series of functions for example) and should have a corresponding test(s) for each function, and should only make changes to that specific function. The entire file if possible (or maybe the header in the case of C++) should still be in the context, but the only part actually being edited should be the 1 specific function, which should be stored in its own place. Then when the agent writes the file to disk, it "compiles" it from those chunks into the final text file that is written. The agent should not have the power to rewrite the entire file at once and re-save it. Otherwise that situation prevents it from systematically working through the problem set, and instead causes it to loop endlessly starting over at the beginning with a rewrite and losing work that it had already done and even debugged. This happened tonight with Claude-Sonnet-3.5 BTW. But I think I've also seen it with GPT-4o, deepseek-coder and/or qwen2 72b instruct. BTW -- you may already be doing this, I saw hints of something like this in the log. But I still saw the problem, so maybe I just need to understand the code better so I can help prevent this sort of issue from happening.