`az resource list` never ends

Martin-Jia commented 6 months ago

Describe the bug

Hi team, I am facing an issue that when I run the command az resource list, it never ends. The command I ran was like: az resource list --subscription xxx --resource-group xxx. Then I added --debug flag. It seems like the command calls a link to get some resource, and there's also a "nextLink" responded with the body. Then the cli keeps to call the "nextLink" to get some resource and the next "nextLink". After several rounds, the returned resource has been an empty list, but there still be a nextLink, and the command keeps to call it and never ends.

Related command

az resource list --subscription xxx --resource-group xxx

Errors

The command never ends.

Issue script & Debug output

Here's some of the debug info. I redacted some info that I thought might be sensitive:

cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/xxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxxxxxxxxxxxx'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '786db126-eb25-11ee-8910-53d35e007ba1'
cli.azure.cli.core.sdk.policies:     'CommandName': 'resource list'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--subscription --resource-group --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.48.1 azsdk-python-azure-mgmt-resource/22.0.0 Python/3.8.5 (Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.10)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/xxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxxxxxxx HTTP/1.1" 200 425
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '425'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '11998'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'ab43965c-da51-40fd-a2f6-c0c3ff18d21c'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': 'ab43965c-da51-40fd-a2f6-c0c3ff18d21c'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'SOUTHEASTASIA:20240326T040139Z:ab43965c-da51-40fd-a2f6-c0c3ff18d21c'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: 7DCD59E0AB4D47268986961C6481F901 Ref B: MAA201060513027 Ref C: 2024-03-26T04:01:31Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Tue, 26 Mar 2024 04:01:39 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[],"nextLink":"https://management.azure.com/subscriptions/dd94e4df-b58f-4e81-9c5e-bd72a4b2ef9e/resources?%24filter=resourceGroup+eq+%27xxxxxxxxxxxx%27&%24expand=createdTime%2cchangedTime%2cprovisioningState&api-version=2022-09-01&%24skiptoken=xxxxxxxxxxx"}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/xxxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxx'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '786db126-eb25-11ee-8910-53d35e007ba1'
cli.azure.cli.core.sdk.policies:     'CommandName': 'resource list'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--subscription --resource-group --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.48.1 azsdk-python-azure-mgmt-resource/22.0.0 Python/3.8.5 (Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.10)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/xxxxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxxxxxxx HTTP/1.1" 200 433
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '433'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '795c0629-d00b-49c0-9308-6ff8d1c5b5b4'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '795c0629-d00b-49c0-9308-6ff8d1c5b5b4'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'SOUTHEASTASIA:20240326T040148Z:795c0629-d00b-49c0-9308-6ff8d1c5b5b4'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: 052934BACB51451BB0BEEB85C1109E4A Ref B: MAA201060513027 Ref C: 2024-03-26T04:01:40Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Tue, 26 Mar 2024 04:01:48 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[],"nextLink":"https://management.azure.com/subscriptions/xxxxxxxxxxxxxx/resources?%24filter=resourceGroup+eq+%27xxxxxxxxxxxx%27&%24expand=createdTime%2cchangedTime%2cprovisioningState&api-version=2022-09-01&%24skiptoken=xxxxxxxx"}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/xxxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxxxxxxxxxxxx'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '786db126-eb25-11ee-8910-53d35e007ba1'
cli.azure.cli.core.sdk.policies:     'CommandName': 'resource list'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--subscription --resource-group --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.48.1 azsdk-python-azure-mgmt-resource/22.0.0 Python/3.8.5 (Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.10)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/xxxxxxxxxxxx/resources?$filter=resourceGroup%20eq%20%27xxxxxxxxxxx%27&$expand=createdTime%2CchangedTime%2CprovisioningState&api-version=2022-09-01&$skiptoken=xxxxxxxxxx HTTP/1.1" 200 429
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '429'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'b3248c69-1777-4efd-9836-33b29ca4bae8'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': 'b3248c69-1777-4efd-9836-33b29ca4bae8'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'SOUTHEASTASIA:20240326T040157Z:b3248c69-1777-4efd-9836-33b29ca4bae8'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: 1F14892DF02548FC87BFA264BD531489 Ref B: MAA201060513027 Ref C: 2024-03-26T04:01:49Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Tue, 26 Mar 2024 04:01:56 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[],"nextLink":"https://management.azure.com/subscriptions/xxxx/resources?%24filter=resourceGroup+eq+%27xxxxxxx%27&%24expand=createdTime%2cchangedTime%2cprovisioningState&api-version=2022-09-01&%24skiptoken=xxxxxxxxxxxxxxxx"}

Expected behavior

resources should be returned.

Environment Summary

azure-cli 2.48.1 *

core 2.48.1 telemetry 1.0.8

Extensions: azure-devops 0.26.0 connectedk8s 1.6.5 k8s-extension 1.4.3 ml 2.24.0

Dependencies: msal 1.20.0 azure-mgmt-resource 22.0.0

Additional context

No response

azure-client-tools-bot-prd[bot] commented 6 months ago

Hi @Martin-Jia,

2.48.1 is not the latest Azure CLI(2.58.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

yonzhan commented 6 months ago

Thank you for opening this issue, we will look into it.

Martin-Jia commented 6 months ago

updated the cli to 2.58.0. The problem is still there.

winterTTr commented 6 months ago

Meet the similar issue here,

az resource list --resource-type 'Microsoft.ContainerRegistry/registries'

never ends.

It seems like, the first response is return a real list with nextLink, however, if we follow the nextLink again, the response is empty list, but also has a nextLink, so az cli is keeping on asking for more even though the server return empty list but still has a non-empty nextLink.

Such as, after the first response with valid list of data, start from the second call following the nextLink fields, all response from server side is as below:

{"value":[],"nextLink":"https://management.azure.com/subscriptions/<omit sensitive data here>"}

This is

either an az cli bug, which should stop further call when the value is already an empty list
or the bug of az management api, which should not return an non-empty nextLink when there is no more data.

Azure / azure-cli