FEAT: Display Token Usage in the UI

tahierhussain commented 4 months ago

What

API to fetch the token usage data.
UI changes to display the token usage data (format - Total Token Count).

-

Why

The token usage for prompt runs was not being displayed to the user earlier.

-

How

NA

-

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Yes, there are quite a few changes in multiple components, so there is a possibility that this PR may affect other features.

-

Database Migrations

NA

Env Config

NA

Relevant Docs

NA

Related Issues or PRs

https://github.com/Zipstack/unstract-cloud/pull/104

Dependencies Versions

NA

Notes on Testing

NA

Screenshots

Checklist

I have read and understood the [Contribution Guidelines]().

hari-kuriakose commented 4 months ago

@tahierhussain I assume the token usage is in the format <embedding>|<prompt>|<completion>|<total>. Can we please add this to the PR description?

Is the total tokens usage calculation correct? From the screenshot, usage of both prompts seems to be less by 1. e.g. 1 + 808 + 3 = 811 instead of 812
Are we showing a tooltip with explanation of display format on hover over token usage?

tahierhussain commented 4 months ago

@tahierhussain I assume the token usage is in the format <embedding>|<prompt>|<completion>|<total>. Can we please add this to the PR description?

Is the total tokens usage calculation correct? From the screenshot, usage of both prompts seems to be less by 1. e.g. 1 + 808 + 3 = 811 instead of 812

Are we showing a tooltip with explanation of display format on hover over token usage?

@hari-kuriakose I have updated the PR description.

Thanks for the catch. You're right. The calculation is not correct. @gaya3-zipstack @Deepak-Kesavan Looks like the total value is not including the embedding tokens. I have verified it by checking the db. Can you please check this?
Yes, a tooltip is added which explains the format.

gaya3-zipstack commented 4 months ago

@tahierhussain I assume the token usage is in the format <embedding>|<prompt>|<completion>|<total>. Can we please add this to the PR description?

Is the total tokens usage calculation correct? From the screenshot, usage of both prompts seems to be less by 1. e.g. 1 + 808 + 3 = 811 instead of 812

Are we showing a tooltip with explanation of display format on hover over token usage?

@hari-kuriakose I have updated the PR description.

Thanks for the catch. You're right. The calculation is not correct. @gaya3-zipstack @Deepak-Kesavan Looks like the total value is not including the embedding tokens. I have verified it by checking the db. Can you please check this?

Yes, a tooltip is added which explains the format.

@tahierhussain @hari-kuriakose @Deepak-Kesavan On point (1), looking at the token usage logging, total_tokens is a field which seems to be applicable only to LLM tokens and not embedding tokens as shown in the image below. If we need to add the embedding token also to the total then we need to change the SDK code to do so . My understanding is that users are only concerned about LLM token count. So do we really need to change "total count" to "total LLM count" instead?

Deepak-Kesavan commented 4 months ago

@tahierhussain Can you update the screenshot with latest changes?

tahierhussain commented 4 months ago

@tahierhussain Can you update the screenshot with latest changes?

@Deepak-Kesavan The current screenshot is the updated one.

hari-kuriakose commented 4 months ago

@tahierhussain I assume the token usage is in the format <embedding>|<prompt>|<completion>|<total>. Can we please add this to the PR description?

Is the total tokens usage calculation correct? From the screenshot, usage of both prompts seems to be less by 1. e.g. 1 + 808 + 3 = 811 instead of 812

Are we showing a tooltip with explanation of display format on hover over token usage?

@hari-kuriakose I have updated the PR description.

Thanks for the catch. You're right. The calculation is not correct. @gaya3-zipstack @Deepak-Kesavan Looks like the total value is not including the embedding tokens. I have verified it by checking the db. Can you please check this?

Yes, a tooltip is added which explains the format.

@tahierhussain @hari-kuriakose @Deepak-Kesavan On point (1), looking at the token usage logging, total_tokens is a field which seems to be applicable only to LLM tokens and not embedding tokens as shown in the image below. If we need to add the embedding token also to the total then we need to change the SDK code to do so . My understanding is that users are only concerned about LLM token count. So do we really need to change "total count" to "total LLM count" instead?

@gaya3-zipstack @tahierhussain Showing the token usage for LLM alone is good enough. We can show the total alone for now, but maybe a break down can be provided in the tooltip later.