search

aws-samples / bedrock-access-gateway

OpenAI-Compatible RESTful APIs for Amazon Bedrock
MIT No Attribution
124 stars 24 forks source link

Missing number of Input token and output token in output response. #15

Closed kambalaashok4 closed 3 weeks ago

kambalaashok4 commented 1 month ago

when an API request is sent to the bedrock-mistral model below details are missing. "prompt_tokens":0,"completion_tokens":0,"total_tokens":0

Please complete the following information: API used: api/v1/chat/completions model used: mistral.mistral-7b-instruct-v0:2

To Reproduce curl http://albenpoint/api/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer bedrock" -d '{ "model": "mistral.mistral-7b-instruct-v0:2", "messages": [{"role":"user","content":"what is hyperloop"}] }'

Expected behavior Output response must contain a number of input tokens and output tokens, prompt tokens

Screenshots image

daixba commented 1 month ago

unfortunately, the bedrock SDK does not provide any info about the token usage for mistral/mixtral model when using invoke_model so far. Other models are fine.

You can switch on the streaming mode (set stream to true) to get the token usage at this stage.

kambalaashok4 commented 1 month ago

Thanks, daixba. can you please help me to where I can set stream to true(which .py file)

daixba commented 1 month ago

It's a parameter in openAI chat completion API. See https://cookbook.openai.com/examples/how_to_stream_completions for more details.

daixba commented 3 weeks ago

This is now supported with the new bedrock api. Please try it again.