Code replacement often doesn't happen. Code is surrounded by '<source>...</source>' tags with no SEARCH/REPLACE tags

[efunneko]

Issue

Somewhat frequently, I am getting the situation where the model does perform the coding as expected, but it does not actually get replaced in the code files. The output from the model looks like this:

> Look for <inst> and follow the instructions in the tag. Also remove the <inst> tags that you have done the work for            

Certainly! I'll look for the <inst> tag, follow the instructions, and then remove the tag once the work is done. Here's the updated content for the agent_registration.md file:

docs/arch/event_definitions/agent_registration.md
<source>
...code removed...
</source>

I've converted the JSON schema to YAML format as requested in the <inst> tag, and then removed the tag. The content now shows the payload schema in YAML format, which is generally considered more readable and concise than JSON for configuration and schema definitions.
Tokens: 20k sent, 532 received.

This looks like it worked properly, but I think that because there are no SEARCH/REPLACE keywords aider just ignores the change. In this case (maybe always when this happens?) the entire file was printed out inside the tag, so maybe it is expected that there is no SEARCH/REPLACE. However, like I said, no file modifications were done and I had to either ask for it again or manually paste the code into the file.

Version and model info

Aider 0.54.4 Model: Sonnet 3.5

[danielbowne]

+1 for this issue using Aider 0.54.4 Model: Sonnet 3.5. Seems to happen more often as of recent releases.

[andrewsgeller]

Agreed. I use claude3.5 via Anthropics chat and often compare results vs. the same model with aider and get vastly different results.

Claude 3.5 sonnet chat is thorough and asks me to consider and take into account other components and concepts.

The a claude-3.5 sonnet via aider is much more transactional and matter of fact. Hard to describe. It makes choices based on only what’s asked.

One thing to consider is to come up with a way to take the query input into aider and use the LLM to reformat the prompt to better suit the chatbot functionality and logic.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: HomeBrewAdmin @.> Sent: Friday, August 30, 2024 7:31 AM To: paul-gauthier/aider @.> Cc: Subscribed @.***> Subject: Re: [paul-gauthier/aider] Code replacement often doesn't happen. Code is surrounded by '...' tags with no SEARCH/REPLACE tags (Issue #1223)

+1 for this issue using Aider 0.54.4 Model: Sonnet 3.5. Seems to happen more often as of recent releases.

— Reply to this email directly, view it on GitHubhttps://github.com/paul-gauthier/aider/issues/1223#issuecomment-2320926428, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJ65MRUS5NCZXV7MITX4ZXTZUBJXNAVCNFSM6AAAAABNLCMU3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRQHEZDMNBSHA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[efunneko]

I am now running the bleeding edge, as of yesterday, (0.54.7dev) and I am really seeing this problem a lot. It does appear to only be when a new file is being created or if the whole file is being replaced. I have had to resort to digging through the chat output and manually cutting and pasting the text into the appropriate file.

As for @andrewsgeller's comment, while totally off-topic, I do agree that the Sonnet output using their web-chat is really really good. Their engineers have created a really good prompt that gets the most out of the model. Happily, they have also officially released this prompt to the public, so there might be some good learnings there for getting the most out of Sonnet

[efunneko]

One more comment on this... When I explicitly including in my prompt that it should be using SEARCH/REPLACE directives in its output, it does do this for files that are newly created. However, despite having the file path right before the block, the new code was inserted into an existing file. The SEARCH block was empty, so perhaps it just matched the first file?

[paul-gauthier]

There is a recent change that may help with this. The change is available in the main branch. You can get it by installing the latest version from github:

aider --install-main-branch

# or...

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, let me know if it works better for you.

[efunneko]

Thanks. I have installed it and will try it out.

I think I have an issue, though. I launch it like this:

aider --model-metadata-file ~/claude_3_5_sonnet.json --model openai/claude-3-5-sonnet 

It uses a Sonnet 3.5 model, hosted on AWS, running through a LiteLLM proxy. So this gives it an OpenAI API, but the model is Anthropic. We have a metadata file (below) that controls how it behaves. However now, I don't think this is working anymore.

Metadata file:

{
    "claude-3-5-sonnet": {
        "max_tokens": 4096,
        "max_input_tokens": 200000,
        "max_output_tokens": 4096,
        "input_cost_per_token": 0.000003,
        "output_cost_per_token": 0.000015,
        "litellm_provider": "anthropic",
        "mode": "chat",
        "supports_function_calling": true,
        "supports_vision": true,
        "tool_use_system_prompt_tokens": 159,
        "supports_assistant_prefill": true
    }
}

And the warning on startup:

Model openai/claude-3-5-sonnet: Unknown context window size and costs, using sane 
defaults.
Did you mean one of these?
- anthropic/claude-3-5-sonnet
- claude-3-5-sonnet
For more info, see: https://aider.chat/docs/llms/warnings.html

Has something changed in this area?

[paul-gauthier]

Nothing has changed other than it pauses now to make sure you see the warning. Did you proceed and try to chat?

[paul-gauthier]

But I do note that you run --model openai/claude-3-5-sonnet but your metdata json has the key claude-3-5-sonnet and litellm_provider = anthropic.

Those all don't seem to agree with each other. I would expect the key and the --model to be the same. Or the --model should be litellm_provider/key.

[efunneko]

I might have a slightly unusual configuration. We have an independent LiteLLM instance running in our company that we use to abstract models and provide individual keys. Since we are using that, all of our usage of any model uses an OpenAI client-side API to talk to LiteLLM.

I used the aider.chat guide (https://aider.chat/docs/llms/openai-compat.html) to configure it to run like this, which is where the 'openai/' prefix on the model name comes from. We made the metadata file to hopefully tell aider that we are actually using a Sonnet model, even though the API is OpenAI. This seemed to work, since it wasn't complaining about not knowing what the context window size was and how much it costs to run the model. Now, it doesn't seem to like it.

If you like, I can raise another issue for this, since this is really off topic...

[paul-gauthier]

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.