Closed Hoekz closed 5 months ago
Thanks for the update. Taking a look
Another file called handler
in message
package also needs to get updated for counting the number of tokens from streaming responses.
For the anthropic text completion, we store the content of streaming responses in a field called content
and do the token counting asynchronously. You can test by checking whether the event generated by the streaming request contains the right completion_token_count
inside postgresql
do you guys need bedrock anthropic integration as well? if yes, i can prioritize it
Anthropic's documentation indicates that you don't need to count tokens while streaming messages since they include the token counts as part of the output unlike OpenAI...is setting the completion token count here not what matters?
We don't need Bedrock support at this time, we use BricksLLM as a sort of Sandbox / Playground for developers rather than for any actual applications.
Anthropic's documentation indicates that you don't need to count tokens while streaming messages since they include the token counts as part of the output unlike OpenAI...is setting the completion token count here not what matters?
We don't need Bedrock support at this time, we use BricksLLM as a sort of Sandbox / Playground for developers rather than for any actual applications.
I see. Ignore what I said. I will proceed to testing the functionality
released a new docker image v1.14.0 for this
Wanted support for the new Anthropic models on our on-prem version of BricksLLM. This seems to be functioning correctly to support the
/messages
endpoint that allows access to the Claude 3 models.