search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
9.3k stars 840 forks source link

[Feature Request] Add token metrics to tracing for Non-OpenAI models #3567

Open jomalsan opened 2 months ago

jomalsan commented 2 months ago

Is your feature request related to a problem? Please describe. I am currently running experiments (flex flows) with Phi-3 mini and want to track the number of prompt and completion tokens as part of my experiments. The tracing view that shows tokens per call and total number of tokens for OpenAI is great, and I'd love to see an easy way to integrate other models with this.

Describe the solution you'd like A clear method or documentation on how to track token consumption for calls to non-OpenAI models like Phi-3. In this case, I think it is expected that the application code must calculate the number of tokens, but there should be a helper method for formatting that token information into the proper format and sending it to the tracing UI.

Describe alternatives you've considered Calculating the token usage as part of the flow and adding them as fields to the flow output response. This way they will show up as columns (and I can use an aggregate method to summarize), but it also exposes that information to any consuming application when I eventually deploy.

brynn-code commented 2 months ago

Thanks for reaching us, we will keep this item open as this requires long-term workload, will update the item if any follow up on this issue.