search

aws-samples / bedrock-access-gateway

OpenAI-Compatible RESTful APIs for Amazon Bedrock
MIT No Attribution
124 stars 24 forks source link

In Server-Sent Events responses, there is no token usage. #8

Closed lishiyu closed 1 month ago

lishiyu commented 2 months ago

In Server-Sent Events responses, there is no token usage.

daixba commented 2 months ago

Hello @lishiyu , thanks for raising the issue.

This will be a low priority item. The purpose of this repo is to help you quickly explore the Bedrock models capability, not all OpenAI API features will be supported.

The recommendation for you is to use Bedrock SDK instead if you need to track the token usage. Or you can do full customization on your own based on this.

daixba commented 1 month ago

Hello @lishiyu The usage info is now added to the streaming response. Please follow the upgrade steps to try the new features.

Make sure you have added "stream_options": {"include_usage": true} parameter to the request as per OpenAI doc.

daixba commented 1 month ago

Close for no updates