Open zmoog opened 4 months ago
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jlichwa @RandalliLama @schaabs.
Hi @zmoog! I think your issue will be fixed by setting the QueryResourcesOptions.Top
field to a higher number. If a filter is specified, the service defaults to 10 records to retrieve per resource ID in the request. This is probably the cause of your throttling issues.
I repo'ed your code locally, and when I set Top
to a higher number, the TimeSeries was consistent between the individual and group query.
options := azmetrics.QueryResourcesOptions{
Aggregation: ptr("Count"),
StartTime: ptr("2024-04-16T07:18:13.001Z"),
EndTime: ptr("2024-04-16T07:19:13.001Z"),
Filter: ptr("ActivityType eq '*' AND ActivityName eq '*' AND StatusCode eq '*' AND StatusCodeClass eq '*'"),
Interval: ptr("PT1M"),
Top: to.Ptr(int32(50)),
}
Hi @zmoog. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.
@gracewilcox apologies if I'm missing something obvious, but:
So why should we need to increase Top if there are fewer than 10 time series per resource? And given that Top should apply per resource, why does it matter if they're queried in bulk vs. one at a time?
@gracewilcox, thank you for replying!
The QueryResourcesOptions.Top
plays a big role in the number of records in the QueryResources()
response.
As you said, the Top
option documentation reports:
The maximum number of records to retrieve per resource ID in the request. Valid only if the filter is specified. Defaults to 10.
However, it seems more like an option that applies to the whole batch and not per resource ID.
In my test case, I have:
All these key vaults are unused (I create them for testing), so I only get two records per resource because we only have two unique combinations of dimension values:
If I set Top = 2
I only get two records for the whole batch:
----------------------------------------------------
Querying resources as a GROUP
----------------------------------------------------
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv1-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv2-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv3-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv4-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv5-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv6-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv7-oeefrt7tlykau
timeseries 1
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv8-oeefrt7tlykau
timeseries 1
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv9-oeefrt7tlykau
timeseries 0
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv10-oeefrt7tlykau
timeseries 0
So I need at least Top = 20 to get all the records.
However, when the resources are used, the number of unique combinations of dimension values varies greatly.
For example, if I start using one one the key vaults, I get 4-5 records instead of 2 for each resource ID:
----------------------------------------------------
Querying resources as a GROUP
----------------------------------------------------
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv1-oeefrt7tlykau
timeseries 5
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv2-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv3-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv4-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv5-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv6-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv7-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv8-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv9-oeefrt7tlykau
timeseries 4
/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/resourceGroups/mbranca-azmetrics-test/providers/Microsoft.KeyVault/vaults/kv10-oeefrt7tlykau
timeseries 4
I guess it's impossible to calculate the exact number of records because it depends on the unique combinations of the dimensions, which vary depending on the Azure service.
@gracewilcox, what strategy do you recommend for setting the Top
value, and what's the maximum number allowed?
Hi @axw and @zmoog! Thank you for the detailed replies. The Top
value is supposed to set the maximum records per resource ID, and as you discovered, it's currently not.
The service team is aware of the issue and is currently deploying a fix. Will let you know as soon as the bug is fixed. Thank you for your patience!
Hi @zmoog, since you haven’t asked that we /unresolve
the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve
to reopen the issue.
/unresolve
Hi $axw, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzmonActionG @AzmonAlerts @AzMonEssential @AzmonLogA @dadunl @SameergMS.
If I run the tests at https://gist.github.com/zmoog/fcede6fcbe5ba11f9275c40a58eea38d I still get the same result.
@gracewilcox, was the service updated with the fix? Does the fix require a new api-version?
@ToddKingMSFT, do you have guidance for this scenario?
Bug Report
github.com/Azure/azure-sdk-for-go/sdk/monitor/query/azmetrics v1.0.0
Context
I am collecting metrics for 10 Microsoft.KeyVault/vaults.
What happened?
If I call
azmetrics.QueryResources()
with a batch of 1-3 resources, I get the same data points I see on Azure Portal.However, I don't get the same values I see in Azure Portal if I try to get metrics values for the same resource in a batch request with 4+ resources.
For example, with a batch of 1-3 resources, I always get two time series values for each resource. Starting with batches of 4+ resources, the number of time series values in the response varies at each request (0-2).
In the following example, I collect the metrics values in two ways:
I get a different number of time series, depending if the same resource is in a batch of 1-3 or 4+ resources.
I repeated this test multiple times. The values for the "SINGLE" case never changed, while the values for the "GROUP" case changed on every call.
What did you expect or want to happen?
For the same resource, I expect
azmetrics.QueryResources()
to always return the same values, whether it's the only resource ID in the batch or one of the 50 supported resource IDs.How can we reproduce it?
I created the gist https://gist.github.com/zmoog/fcede6fcbe5ba11f9275c40a58eea38d with:
azmetrics.QueryResources()
Anything we should know about your environment.
Additional info
I see the same behavior when calling the API endpoint using cURL. See https://github.com/zmoog/public-notes/issues/81 for more details.
Questions: