search

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
21.31k stars 3.13k forks source link

.Net: Divided metering #7596

Open markwallace-microsoft opened 1 month ago

markwallace-microsoft commented 1 month ago

Discussed in https://github.com/microsoft/semantic-kernel/discussions/7583

Originally posted by **imranshams** July 31, 2024 Hi SK Community, I'm facing a requirement regarding divided counter by tags or something like that. We are using Azure OpenAI and we want to charge our users based on their Token usage or session based, by default we could find total token usage in Azure AI Studio in metric section. I was just reading the telemetry doc here: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/docs/TELEMETRY.md Our approach is creating Kernel in each session start and inject with DI. It will be great if we could assign tags to Kernel in build step and by gathering metrics they will be assigned to counters, so we can track them in billing & metrics in Azure Portal & logs. Microsoft.SemanticKernel.Connectors.OpenAI - available metrics for OpenAI functionality: - semantic_kernel.connectors.openai.tokens.prompt (Counter) - number of prompt tokens used. - semantic_kernel.connectors.openai.tokens.completion (Counter) - number of completion tokens used. - semantic_kernel.connectors.openai.tokens.total (Counter) - total number of tokens used. Thank you so much for your time and effort.
evchaki commented 3 weeks ago

@TaoChenOSU can you take a look at this? Can we add the ability to add custom Tag to our telemetry hooks to support this and other use cases.

TaoChenOSU commented 3 weeks ago

This is doable. We need the following sub-features to satisfy this requirement:

  1. Allow users to assign tags to the kernel, or at least assign an identifier to a kernel (it will be the user's responsibility to guarantee uniqueness). This has not been implemented.
  2. Pass the kernel to the ChatCompletionService, TextGenerationService, etc. This has been implemented.
  3. Associate tags, or the kernel identifier to the measurements. This has not been implemented but should be fairly straightforward.

The question comes down to how we would like to give a kernel instance an identifier or allow users to tag custom data to a kernel.

Depending on your setup, there is another option that doesn't require us to make changes to the Kernel source code. This approach involves creating unique resourceBuilders for your sessions: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Demos/TelemetryWithAppInsights/Program.cs#L43. The service name of the resource will be automatically associated to the measurements created by the MeterProvider associated with the resource. In Application Insights, it will be the cloud_RoleName property: image

TaoChenOSU commented 2 weeks ago

I will draft a short ADR for the first option proposed above.