search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.63k stars 2.84k forks source link

modeldeploymentname is `__Empty` #37982

Open DanielHabenicht opened 1 month ago

DanielHabenicht commented 1 month ago

Describe the bug The modeldeploymentname is empty for certain deployed models. Namely DALLE3.Standard.Low and Whisper Image

To Reproduce Query the metrics of the specified models: 

ids = ["your/id"]
response = await self.metrics_clients[location].query_resources(
                resource_ids=ids,
                metric_namespace="Microsoft.CognitiveServices/accounts",
                metric_names=[
                    "TokenTransaction",
                ],
                filter="ModelDeploymentName eq '*' and FeatureName eq '*'",
                timespan=(start_time, end_time),
                granularity=timedelta(days=1),
                aggregations=[MetricAggregationType.TOTAL],
            )

Expected behavior It should contain the model name as specified during the deployment, instead of empty.

Additional context I can provide the exact resource id via PN.

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @gulopesd @Haiying-MSFT @jairmyree @joshfree @KarishmaGhiya @KevinBlasko @kurtzeborn @nisha-bhatia @pvaneck @sarangan12 @scottaddie @srnagar @ToddKingMSFT.

lzchen commented 1 month ago

@xiangyan99

I do not own `azure-monitor-query, please reassign.

pvaneck commented 1 month ago

Hey @DanielHabenicht. The azure-monitor-query SDK only outputs what the monitor metrics API returns, so it's likely a service-side issue. If you were to view the OpenAI metrics in the Azure Portal, does it show the same? I'm not sure what's up, but it seems like some of the dimensions aren't being populated/leveraged for the TokenTransaction metric. Couldn't find much online regarding the issue (only really this question for something similar for Cosmos metrics).

You'd probably get better eyes on this by opening a question on the Microsoft Q&A website or opening a support request (can do this by going to the "Help + support" option in the left-hand menu of your OpenAI resource in the Azure Portal).

DanielHabenicht commented 1 month ago

In the Azure Portal the TokenTransaction Metric is not available :/ Image

I will ask the support directly and update this issue accordinly.

github-actions[bot] commented 1 month ago

Hi @DanielHabenicht. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

pvaneck commented 1 month ago

Based on these docs TokenTransaction corresponds to Processed Inference Tokens. After that, can apply splitting on ModelName and ModelDeploymentName

DanielHabenicht commented 1 month ago

Based on these docs TokenTransaction corresponds to Processed Inference Tokens. After that, can apply splitting on ModelName and ModelDeploymentName

Yes, somehow I must have overlooked that. It shows up in the Azure Portal as well: Image Image