Open xcke opened 2 weeks ago
Thank you for filing this issue.
cat ~/.aider.models.yml
The correct filename to override model configurations is .aider.model.settings.yml
see here -> https://aider.chat/docs/config/adv-model-settings.html#model-settings
I fixed the name of the file, and also updated to include supported configurations.
However, still getting Token limit errors. Even if lowering the max_tokens, the effect is the same:
- cache_control: true
edit_format: diff
editor_edit_format: editor-diff
editor_model_name: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
examples_as_sys_msg: true
extra_params:
extra_headers:
anthropic-beta: prompt-caching-2024-07-31
# anthropic-beta: max-tokens-3-5-sonnet-2024-07-15
max_tokens: 4092
name: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
reminder: user
use_repo_map: true
weak_model_name: openai/gpt-4o-mini
Is there a way to add prefill support here?
You need to use the model metadata json file to indicate that it supports prefill.
Thanks @paul-gauthier . I think this solved my issue:
{
"bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0": {
"max_tokens": 4192,
"max_input_tokens": 128000,
"max_output_tokens": 4192,
"input_cost_per_token": 0.000003,
"output_cost_per_token": 0.000015,
"litellm_provider": "bedrock",
"supports_assistant_prefill": true,
"supports_prompt_caching": true,
"mode": "chat"
}
}
Hi,
We have implemented a lot of improvements regarding Sonnet 3.5, however when using it via Bedrock (that is the older Sonner 3.5 version) I keep running into the Token Limit error.
I have tried to use model config:
I know AWS is not using the latest model version, so by default it has 4k output tokens.
Is there a good workaround or model configuration option to get ahead of this?