search

aws-samples / bedrock-claude-chat

AWS-native chatbot using Bedrock + Claude (+Mistral)
MIT No Attribution
688 stars 232 forks source link

[Feature Request] Refactoring with Bedrock Converse API #347

Open hustshawn opened 3 weeks ago

hustshawn commented 3 weeks ago

Describe the solution you'd like

Amazon Bedrock just released the converse API, which provides a consistant way to invoke models and genereate response.

Why the solution needed

Currently the project using Anthropic client for Claude models, and boto3 bedrock client for Mistral models. Both of them are way different in handling invocation and response parsing. It would be ideal to leverage the Converse API to make the code base clean.

Additional context

Converse API: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html

Implementation feasibility

Are you willing to discuss the solution with us, decide on the approach, and assist with the implementation?

statefb commented 3 weeks ago

Thank you for your suggestion! While I think it's a good suggestion, I'm concerned that with innovative updates like the multimodal chat in claude3, the Converse API may lag behind in terms of support.

Currently, since there are only two models, Claude and Mistral, for model invocation, we're not having any maintenance issues. Particularly for streaming, we recently refactored it (Also unit test can be found here), so we don't see much value in migrating. However, if there is a significant need to add models other than Mistral, we may consider adopting the Converse API as you suggested. How do you think?

hustshawn commented 3 weeks ago

It looks a really neat refactoring, and I was not aware of it before. Thank you for mention it.

Yeah, I understand you may not consider to prioritize it at the moment. I can think about potential timeline to go with the Converse API could be either the bedrock invoke API is deprecated, or there is demand to support extra models like Llama3.

I suppose it's good to keep this ticket open, and see if people have same interest or requirement on extra models support.

statefb commented 1 day ago

The converse API is updated: supports document

https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html image image

statefb commented 1 day ago

Idea notebook