sigoden / aichat

All-in-one AI CLI tool featuring Chat-REPL, Shell Assistant, RAG, AI tools & agents, with access to OpenAI, Claude, Gemini, Ollama, Groq, and more.
Apache License 2.0
3.93k stars 265 forks source link

max_output_tokens option from model in config.yaml is ignored when model is selected #641

Closed henryprecheur closed 3 months ago

henryprecheur commented 3 months ago

Describe the bug

I added Claud 3.5 Sonnet to my aichat configuration and I get the following error:

Failed to get answer

Caused by:
    max_tokens: Field required (type: invalid_request_error)

To Reproduce

config.yaml

model: claude:claude-3-5-sonnet-20240620
clients:
- type: claude
  api_key: sk-ant-api03-*
  models:
    - name: claude-3-5-sonnet-20240620
      max_input_tokens: 200000
      max_output_tokens: 4096
      require_max_tokens: true
      input_price: 3
      output_price: 15
      supports_vision: true
      supports_function_calling: true

Start aichat and try to use model:

$ aichat
Welcome to aichat 0.18.0
Type ".help" for additional help.
> .info
model               claude:claude-3-5-sonnet-20240620
max_output_tokens   -
temperature         -
top_p               -
function_calling    false
compress_threshold  4000
dry_run             false
save                false
save_session        -
highlight           true
light_theme         false
wrap                no
wrap_code           false
auto_copy           false
keybindings         emacs
prelude             -
config_file         /home/henry/llm/aichat/config.yaml
roles_file          /home/henry/llm/aichat/roles.yaml
messages_file       /home/henry/llm/aichat/messages.md
sessions_dir        /home/henry/llm/aichat/sessions
functions_dir       /home/henry/llm/aichat/functions

> Test
Failed to get answer

Caused by:
    max_tokens: Field required (type: invalid_request_error)
> .set max_output_tokens 4096

> Test
I'm here and ready to help! What would you like assistance with?
> .info
model               claude:claude-3-5-sonnet-20240620
max_output_tokens   4096 (current model)
temperature         -
top_p               -
function_calling    false
compress_threshold  4000
dry_run             false
save                false
save_session        -
highlight           true
light_theme         false
wrap                no
wrap_code           false
auto_copy           false
keybindings         emacs
prelude             -
config_file         /home/henry/llm/aichat/config.yaml
roles_file          /home/henry/llm/aichat/roles.yaml
messages_file       /home/henry/llm/aichat/messages.md
sessions_dir        /home/henry/llm/aichat/sessions
functions_dir       /home/henry/llm/aichat/functions

Expected behavior

The configured max_output_tokens option is used by aichat when submitting the request.

Environment (please complete the following information):

sigoden commented 3 months ago

see https://github.com/sigoden/aichat/pull/598#issuecomment-2167032180