az quota update request fails with "Request Header Fields Too Large" error

[alexp-openai]

Describe the bug

I'm running a quota update command from the cli, like this:

az quota update --scope "/subscriptions/37537258-29ed-4742-87d2-e909358c01fc/providers/Microsoft.Compute/locations/southcentralus" --resource-name "standardDCSv2Family" --limit-object value=7 --resource-type dedicated

And I'm getting an error message "Operation returned an invalid status 'Request Header Fields Too Large'".

I tried running it with --debug flag, and all of the HTTP headers look normal-size. The authorization header is hidden, so maybe it is too long.

Other az commands, like az quota list, work fine.

Related command

az quota update

Errors

Operation returned an invalid status 'Request Header Fields Too Large'

Issue script & Debug output

msal.telemetry: Generate or reuse correlation_id: 900fd643-0e28-4aab-8bf0-98a66d55f1e4
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/37537258-29ed-4742-87d2-e909358c01fc/providers/Microsoft.Compute/locations/southcentralus/providers/Microsoft.Quota/quotas/standardDv3Family?api-version=2023-02-01'
cli.azure.cli.core.sdk.policies: Request method: 'PATCH'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'Content-Length': '141'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': 'd2e43064-2ccd-11ef-8d01-9ab805ee86ed'
cli.azure.cli.core.sdk.policies:     'CommandName': 'quota update'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--scope --resource-name --limit-object --resource-type --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.61.0 (HOMEBREW) azsdk-python-core/1.28.0 Python/3.11.9 (macOS-14.5-x86_64-i386-64bit)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"properties": {"limit": {"limitObjectType": "LimitValue", "value": 0}, "name": {"value": "standardDv3Family"}, "resourceType": "dedicated"}}
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "PATCH /subscriptions/37537258-29ed-4742-87d2-e909358c01fc/providers/Microsoft.Compute/locations/southcentralus/providers/Microsoft.Quota/quotas/standardDv3Family?api-version=2023-02-01 HTTP/1.1" 431 0
cli.azure.cli.core.sdk.policies: Response status: 431
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-resource-requests': '23'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '4db521fc-6d1b-4d59-a509-4425bf93dc08'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '4db521fc-6d1b-4d59-a509-4425bf93dc08'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'WESTUS:20240617T172001Z:4db521fc-6d1b-4d59-a509-4425bf93dc08'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: F692E25BCBA946C98D4F7EA29C8C7749 Ref B: SJC211051204045 Ref C: 2024-06-17T17:20:01Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Mon, 17 Jun 2024 17:20:01 GMT'
cli.azure.cli.core.sdk.policies:     'Content-Length': '0'

Expected behavior

the command should work without error

Environment Summary

azure-cli                         2.61.0

core                              2.61.0
telemetry                          1.1.0

Extensions:
quota                              1.0.0

Dependencies:
msal                              1.28.0
azure-mgmt-resource               23.1.1

Python location '/usr/local/Cellar/azure-cli/2.61.0/libexec/bin/python'
Extensions directory '/Users/xxx/.azure/cliextensions'

Python (Darwin) 3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[alexp-openai]

Any updates?

[ajhous44]

Also looking for updates here

[ajhous44]

@yonzhan

[ajhous44]

Python SDK also throws same 431 error

[pa-openai]

Hi, are there any updates on this issue?

[alexp-openai]

Created support request #2407240010005397 in Azure Portal.

[ajhous44]

@alexp-openai I opened a request too a few days ago. Being tossed around internally at MSFT and am getting updates every 2 days but no real update. Got on a call with an engineer at one point, and it worked fine for them but not me, but referred them here to show it's not just me. We shall see who wins the support race.... Can we agree to update each other here? 😉

[alexp-openai]

Just to add context, we do have several hundreds of subscriptions visible for each user. Maybe that affects the size of authentication header.

[ajhous44]

But we specify the subscription / scope in the call, no? Even in the try it now portion of https://learn.microsoft.com/en-us/rest/api/reserved-vm-instances/quota/update?view=rest-reserved-vm-instances-2022-11-01&tabs=HTTP#code-try-0 it's denied.

Would be surprised if # subs affected request headers, but then again... maybe not. Also possible there was an update to token length or something by a diff team outside of the team that works on the quota api and they're not aware.

[ajhous44]

@sima-zhu @xiangyan99

[alexp-openai]

We found out that users on our team has very large access tokens in ~/.azure/msal_token_cache.json file. One of them is 10109 bytes long.

[ajhous44]

@alexp-openai What was your resolution?

[jsntcy]

@rahuls-microsoft, could you please help take a look as it seems the error is from server side? Both CLI and SDK return the same error.

[ajhous44]

@zhenlan saw you helped resolve similar issue under https://github.com/Azure/AppConfiguration/issues/233. Not sure if you can be of assistance here as well?

[alexp-openai]

@ajhous44 , I haven't found a working solution that can be use from CLI or API. As a workaround, I edit quotas manually through Azure Portal UI.

[rahuls-microsoft]

We found that the user who using the API had membership to multiple groups (164) and which were included in the header, which caused the issue header too big. I would recommend using an APID, which has Limited/Required Group Membership not to all the Groups. I think, it was temporary test by a User, that's why you got this error.

[alexp-openai]

@rahuls-microsoft , thanks for looking into this and suggesting the workaround! I am indeed a member of many groups, but they are used for different purposes, so such large number of groups is not an accidental mistake.

Is it possible to increase header limit for the quota API to a larger value? I think I was able to use other az cli command successfully. Quota increments through the Azure Portal UI also completes successfully, despite sending a very large authentication token there as well(we used Chrome developer tools to see the headers being sent to quota API from the browser)

[pa-openai]

Also worth noting: I have a colleague for whom the quota update does work, and he's in more groups than I am. So it seems like the number of groups might not be strictly the issue?

[rahuls-microsoft]

As Per Logs the error is - 431 Request Header Fields Too Large The current Limit is already 32K. Quota RP requests Group Information to get the user's group membership to validate certain scenarios, which is provided by ARM. Other RPs may not need it, so they won't get impacted. The header is the Sum of all the header fields, so it might cross the limit in some cases. It seems like it's just on the boundary limits, so it works with portal and not with Az. Can you try Curl or PowerShell, just to validate, if Az has any special headers, which is crossing the header size Limit of 32K. As you mentioned that it works through portal, so there might be some extra headers added by as client, which might be hitting the limit. Meanwhile I will look into a resolution, but that would take time to implement and deploy, if possible. Let's find a working solution to unblock you and work to increase the header limits, if possible. Thanks for your help in investigating the issue.