search

simonw / llm-claude-3

LLM plugin for interacting with the Claude 3 family of models
Apache License 2.0
254 stars 23 forks source link

Sonnet 3.5: Now with 8192 reasons to love it #10

Closed ghostofpokemon closed 2 months ago

ghostofpokemon commented 3 months ago

Hey Simon,

I've made some updates to support the new 8192 token limit for Claude 3.5 Sonnet. Here's what's changed:

  1. Updated max_tokens handling in ClaudeMessages.execute():

    • Set default to 8192 for Claude 3.5 Sonnet
    • Maintained 4096 for other models
    • Uses prompt.options.max_tokens if specified, falling back to these defaults

  2. Added the required beta header for Claude 3.5 Sonnet:

    {"anthropic-beta": "max-tokens-3-5-sonnet-2024-07-15"}

  3. Changed max_tokens default in ClaudeOptions to None for more flexibility

These changes maintain backwards compatibility while enabling the new token limit for Claude 3.5 Sonnet. I've tested the changes and they appear to be working as expected.

Let me know if you need any further information or if you'd like me to make any adjustments.

Thanks for considering this PR!

𝕏 @NickMystic_

gbid commented 2 months ago

Really appreciate the work put into the llm cli. I also direly need the increased max_tokens implemented in this PR. Is there any plan on merging this PR in the near future?

simonw commented 2 months ago

My apologies, I shipped this feature just now without noticing your PR!

This is now released in 0.4.1.