Hitting the Token Output Limit with Sonnet

[Th0rGer]

Issue

When using the openrouter Sonnet Model it still says to me that i have a max token output limit of 4096.

In openrouter is statet that the limit is 8192 Tokens.

Version and model info

Aider v0.50.2-dev

[Th0rGer]

Also Infitiy output on sonnet is not enabled when using the openrouter model.

[paul-gauthier]

Thanks for trying aider and filing this issue.

When reporting problems, it is very helpful if you can provide:

• Aider version
• LLM model you are using
• Any stack traces or error messages
• A description of what you were doing when the error happened.

Including the “announcement” lines that aider prints at startup is an easy way to share some of this helpful info.

Aider v0.37.1-dev
Models: gpt-4o with diff edit format, weak model gpt-3.5-turbo
Git repo: .git with 243 files
Repo-map: using 1024 tokens

[Th0rGer]

with openrouter models: aider --cache-prompts --model openrouter/anthropic/claude-3.5-sonnet

Aider v0.51.1 Main model: openrouter/anthropic/claude-3.5-sonnet with diff edit format Weak model: openrouter/anthropic/claude-3-haiku-20240307 Git repo: .git with 22 files Repo-map: using 1024 tokens, files refresh VSCode terminal detected, pretty output has been disabled. Use /help for help, run "aider --help" to see cmd line args

with claude direct models:

aider --cache-prompts

Aider v0.51.1 Main model: claude-3-5-sonnet-20240620 with diff edit format, prompt cache, infinite output Weak model: claude-3-haiku-20240307 Git repo: .git with 22 files Repo-map: using 1024 tokens, files refresh VSCode terminal detected, pretty output has been disabled. Use /help for help, run "aider --help" to see cmd line args

as with the openrouter model the caching and infinite output is not enabled.

[paul-gauthier]

The fix is available in the main branch. You can get it by installing the latest version from github:

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, let me know if it works better for you.

[KoStard]

Hello @paul-gauthier,

I frequently hit output token limit with Bedrock Claude 3.5. I was going through the aider codebase hoping to contribute, but I don't see any pointers of this functionality.

Can you please share if there is a fix for this issue or how can I contribute to it?

Thanks, Ruben

[paul-gauthier]

To enable infinite output, the model metadata needs to have supports_assistant_prefill = true. I am not sure if Bedrock actually supports that though.

https://aider.chat/docs/config/adv-model-settings.html

But openrouter Sonnet does in recent aider versions:

$ aider --model openrouter/anthropic/claude-3.5-sonnet

───────────────────────────────────────────────────────────────────────────────────────────────
Aider v0.54.13.dev2+g5fe85f72.d20240902
Main model: openrouter/anthropic/claude-3.5-sonnet with diff edit format, infinite output

[paul-gauthier]

I'm going to close this issue for now, but feel free to add a comment here and I will re-open. Or feel free to file a new issue any time.