RobertCraigie
commented
4 weeks ago Thanks for reporting!
This sounds like an issue with the underlying Anthropic API and not the SDK, so I'm going to go ahead and close this issue.
I recommend joining the community discord or for more details on API support and assistance, see this support article.
Hello,
While the new model seems to have some improved capabilities, it has a very severe issue for me and my API use cases. I use the prompt generator tool on the Anthropic workbench and set the output to be ~3500 tokens. However, no matter what I do, the model generates only 200-300 tokens and then puts a statement in brackets telling me this message would be too long and can generate the rest of the completion in a subsequent message. The exact same prompt used with the previous version of sonnet-3.5 outputs the full output in good alignment with the few shot examples in the original prompt. I have tried altering the system prompt, the user prompt, the prefill prompt, nothing seems to be able to get the model to output in the same format the previous version did 100% of the time.
Messages like these appear after only 200 tokens of generation:
[Continuing with revised character card in next message due to length...]All of the example character cards are only about 1000-1500 tokens, but nothing I do will get it to output that amount. Is there any way that people using the API can have the model generate outputs without having to "click through" over and over again confirming that we want the text to be generated? This is such a serious issue it may require me to move fully back to OpenAI API or continue developing with the old version of sonnet-3.5 for this kind of work. Please do not force developers to have to constantly ask for medium length completions (that are shorter than the models stated maximum output length).