usageMetadata is missing when using generateContent

[marczi]

When calling the generative model with generateContent() then usageMetadata is missing, however, when called with generateContentStream(), then each streamed item contains the usageMetadata object.

The usage info is lost in /src/process_stream.ts, in the aggregateResponses() function which accumulates all candidates, but ignores the usageMetadata. I suppose, in this function, the sum of the used tokens should be also returned in the returned GenerateContentResult.

Environment details

• Programming language: Javascript/TypeScript
• OS: Windows 11
• Language runtime version: Node.js 20.11.0
• Package version: @google-cloud/vertexai:0.2.1

Steps to reproduce

Try the following code. In generateContentStream(), the streamed items print out the usageMetada, but in generateContent() this is missing.

const vertex_ai = new VertexAI({project: 'my-project', location: 'us-central1'});
const model = 'gemini-pro';

// Text model
const generativeModel = vertex_ai.preview.getGenerativeModel({
    model: model,
    generation_config: {
        "max_output_tokens": 2048,
        "temperature": 0.9,
        "top_p": 1
    },
    safety_settings: [],
});

async function generateContentStream() {
    const req = {
        contents: [{role: 'user', parts: [{text: "Say hi"}]}],
    };

    const streamingResp = await generativeModel.generateContentStream(req);

    for await (const item of streamingResp.stream) {
        // usageMetadata will be printed here
        console.log('stream item: ', JSON.stringify(item));
    }

    const result = await streamingResp.response;
    // usageMetadata is missing here
    console.log('aggregated response: ', JSON.stringify(result));
};

async function generateContent() {
    const req = {
        contents: [{role: 'user', parts: [{text: "Say hi"}]}],
    };

    const resp = await generativeModel.generateContent(req);
    // usageMetadata is missing here as well
    console.log('aggregated response: ', JSON.stringify(resp));
};

generateContentStream();
generateContent();

[yyyu-google]

Thank you @marczi I have tested and confirmed that the generateContent api returns usageMetadata but generateContentStream doesn't. I am working on the fix.

[yyyu-google]

this pull request should fix the problem. will close this comment when next release is done.

https://github.com/googleapis/nodejs-vertexai/pull/182

[yyyu-google]

Hi @marczi version 0.5.0 should fix this problem.

[BaraKnostic]

Im also using the generateContent and does not receive any usageMetadata.

request https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent?key=<<KEY>>

{
  "contents": [
    {
      "role": "USER",
      "parts": { "text": "Hello!" }
    },
    {
      "role": "MODEL",
      "parts": { "text": "Argh! What brings ye to my ship?" }
    },
    {
      "role": "USER",
      "parts": { "text": "Wow! You are a real-life priate!" }
    }
  ],
  "safety_settings": {
    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
    "threshold": "BLOCK_LOW_AND_ABOVE"
  },
  "generation_config": {
    "temperature": 0.2,
    "topP": 0.8,
    "topK": 40,
    "maxOutputTokens": 200,
  }
}

response

{
    "candidates": [
        {
            "content": {
                "parts": [
                    {
                        "text": "Aye, that I be! And what be ye after, landlubber?"
                    }
                ],
                "role": "model"
            },
            "finishReason": "STOP",
            "index": 0,
            "safetyRatings": [
                {
                    "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
                    "probability": "NEGLIGIBLE"
                },
                {
                    "category": "HARM_CATEGORY_HATE_SPEECH",
                    "probability": "NEGLIGIBLE"
                },
                {
                    "category": "HARM_CATEGORY_HARASSMENT",
                    "probability": "NEGLIGIBLE"
                },
                {
                    "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
                    "probability": "NEGLIGIBLE"
                }
            ]
        }
    ],
    "promptFeedback": {
        "safetyRatings": [
            {
                "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
                "probability": "NEGLIGIBLE"
            },
            {
                "category": "HARM_CATEGORY_HATE_SPEECH",
                "probability": "NEGLIGIBLE"
            },
            {
                "category": "HARM_CATEGORY_HARASSMENT",
                "probability": "NEGLIGIBLE"
            },
            {
                "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
                "probability": "NEGLIGIBLE"
            }
        ]
    }
}

[happy-qiao]

Hi BaraKnostic

This is Google Vertex AI git repository. This url https://generativelanguage.googleapis.com/v1/models/gemini-pro:generateContent?key=<<KEY>> indicates you are not using the Vertex AI endpoint.

To use Gemini in Vertex AI, you can follow https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal#gemini-beginner-samples-nodejs