search

bricks-cloud / BricksLLM

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
https://trybricks.ai/
MIT License
863 stars 60 forks source link

Add Support for Anthropic Messages #51

Closed Hoekz closed 5 months ago

Hoekz commented 5 months ago

Wanted support for the new Anthropic models on our on-prem version of BricksLLM. This seems to be functioning correctly to support the /messages endpoint that allows access to the Claude 3 models.

spikelu2016 commented 5 months ago

Thanks for the update. Taking a look

spikelu2016 commented 5 months ago

Another file called handler in message package also needs to get updated for counting the number of tokens from streaming responses.

For the anthropic text completion, we store the content of streaming responses in a field called content and do the token counting asynchronously. You can test by checking whether the event generated by the streaming request contains the right completion_token_count inside postgresql

spikelu2016 commented 5 months ago

do you guys need bedrock anthropic integration as well? if yes, i can prioritize it

Hoekz commented 5 months ago

Anthropic's documentation indicates that you don't need to count tokens while streaming messages since they include the token counts as part of the output unlike OpenAI...is setting the completion token count here not what matters?

We don't need Bedrock support at this time, we use BricksLLM as a sort of Sandbox / Playground for developers rather than for any actual applications.

spikelu2016 commented 5 months ago

Anthropic's documentation indicates that you don't need to count tokens while streaming messages since they include the token counts as part of the output unlike OpenAI...is setting the completion token count here not what matters?

We don't need Bedrock support at this time, we use BricksLLM as a sort of Sandbox / Playground for developers rather than for any actual applications.

I see. Ignore what I said. I will proceed to testing the functionality

spikelu2016 commented 5 months ago

released a new docker image v1.14.0 for this