Closed marczi closed 6 months ago
Thank you @marczi
I have tested and confirmed that the generateContent
api returns usageMetadata but generateContentStream
doesn't. I am working on the fix.
this pull request should fix the problem. will close this comment when next release is done.
Hi @marczi
version 0.5.0
should fix this problem.
Im also using the generateContent
and does not receive any usageMetadata.
request
https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent?key=<<KEY>>
{
"contents": [
{
"role": "USER",
"parts": { "text": "Hello!" }
},
{
"role": "MODEL",
"parts": { "text": "Argh! What brings ye to my ship?" }
},
{
"role": "USER",
"parts": { "text": "Wow! You are a real-life priate!" }
}
],
"safety_settings": {
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_LOW_AND_ABOVE"
},
"generation_config": {
"temperature": 0.2,
"topP": 0.8,
"topK": 40,
"maxOutputTokens": 200,
}
}
response
{
"candidates": [
{
"content": {
"parts": [
{
"text": "Aye, that I be! And what be ye after, landlubber?"
}
],
"role": "model"
},
"finishReason": "STOP",
"index": 0,
"safetyRatings": [
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"probability": "NEGLIGIBLE"
}
]
}
],
"promptFeedback": {
"safetyRatings": [
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"probability": "NEGLIGIBLE"
}
]
}
}
Hi BaraKnostic
This is Google Vertex AI git repository. This url https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent?key=<<KEY>>
indicates you are not using the Vertex AI endpoint.
To use Gemini in Vertex AI, you can follow https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal#gemini-beginner-samples-nodejs
When calling the generative model with generateContent() then usageMetadata is missing, however, when called with generateContentStream(), then each streamed item contains the usageMetadata object.
The usage info is lost in /src/process_stream.ts, in the aggregateResponses() function which accumulates all candidates, but ignores the usageMetadata. I suppose, in this function, the sum of the used tokens should be also returned in the returned GenerateContentResult.
Environment details
Steps to reproduce
Try the following code. In generateContentStream(), the streamed items print out the usageMetada, but in generateContent() this is missing.