search

firebase / genkit

An open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations. Genkit works with various models and platforms.
Apache License 2.0
727 stars 106 forks source link

[Tooling] Show input/output token count in Dev UI #798

Open apascal07 opened 2 months ago

apascal07 commented 2 months ago

Describe the bug Input/output token counts are tracked but are only visible in GCP. It would be nice to see the input/output counts directly in the trace in Dev UI to get an idea for how expensive the call is during development.

Edit: Token counts are available in the output section of the trace viewer (e.g. on the model action).

  1. For the trace view, we should consider add badging and/or other UI affordances to elevate this information into clear sight.
  2. We could also consider displaying token counts on the model and prompt playgrounds as well.
MichaelDoyle commented 2 months ago

Typically I am able to see these values in my local traces. Can you clarify?

image
apascal07 commented 2 months ago

Ah, I didn't realize. I kind of expect it to appear everywhere latency does (left sidebar, right sidebar), maybe with parent nodes summing child nodes so that you can get a feel for costs at a glance just like you can with latency. WDYT?

MichaelDoyle commented 2 months ago

Definitely - that sounds good to me (badges for tokens, etc). Thanks for clarifying.