vercel / ai

Build AI-powered applications with React, Svelte, Vue, and Solid
https://sdk.vercel.ai/docs
Other
9.39k stars 1.37k forks source link

Prompt Token Usage not accurate when using Anthropic Prompt Caching #2845

Closed ghardin1314 closed 1 week ago

ghardin1314 commented 2 weeks ago

Description

When using the experimental_providerMetadata property for caching prompts with Anthropic provider, the token usage that is reported is inaccurate. The prompt tokens reported are only counting the non-cached part of the prompt, even though there is still (smaller) costs with using the cached section of the prompt.

Code example

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAI } from "@ai-sdk/openai";
import { generateObject } from "ai";

  const anthropicProvider = createAnthropic({
      apiKey: env.ANTHROPIC_API_KEY,
    });

  const model = anthropicProvider.languageModel("...", { cacheControl: true })

  await generateObject({
                       model,
                       schema: ...,
                       messages: [
                       {
                         role: "system",
                         content: ...,
                       },
                       {
                         role: "user",
                         content: ...,
                         experimental_providerMetadata: {
                           anthropic: { cacheControl: { type: "ephemeral" } },
                         },
                       },
                       // Prompt tokens above here are not reported in the usage
                       {
                         role: "user",
                         content: ...
                       },
                      ]

Additional context

No response

lgrammel commented 2 weeks ago

@ghardin1314 the additional information about the cached tokens is returned as well (provider specific): https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic#example-cache-control