How to expose vendor specific usage information

spring-projects / spring-ai

An Application Framework for AI Engineering

https://docs.spring.io/spring-ai/reference/index.html

Apache License 2.0

3.17k stars 788 forks source link

How to expose vendor specific usage information #1407

Open markpollack opened 1 month ago

markpollack commented 1 month ago

OpenAI added a new usage field,

    "usage": {
        "prompt_tokens": 13,
        "completion_tokens": 7,
        "total_tokens": 20,
        "completion_tokens_details": {
            "reasoning_tokens": 0
        }
    },

this should be made available in some way via the org.springframework.ai.chat.metadata.Usage interface

See https://github.com/spring-projects/spring-ai/pull/1394

dafriz commented 1 week ago

OpenAI has a nested audio_tokens usage field in both the prompt_tokens_details and completion_tokens_details so we can't just add them all as key value pairs unless key is prefixed or similar.

https://platform.openai.com/docs/api-reference/chat/object

ThomasVitale commented 1 week ago

The prompt_tokens_details and completion_tokens_details.audio_tokens components are being introduced in this PR: https://github.com/spring-projects/spring-ai/issues/1560 I haven't propagated them up due to the challenges covered in this issue.

Having reasoning_tokens at the top level makes it challenging to support the other fields, especially audio_tokens which is present both for prompt and completion.