Open nerddtvg opened 4 years ago
balag0
commented
4 years ago Nice catch.. I wonder if this is the same behaviour causing https://github.com/projectkudu/AzureResourceExplorer/issues/228
Thanks for debugging this and finding the root cause. Looking forward to your PR :)
nerddtvg
commented
4 years ago The logic is simple. In the end, I question the need for the maximum depth search. This may resolve some issues from #228, but if there are too many API calls to retrieve all subscription IDs, this may not be enough.
I reached out to someone on Slack if they could point me in the right direction on if there is actually a case where the API will return duplicate resources or skiptokens. I really can't see that happening from all the testing and code work I've done with this API in the past.
nerddtvg
commented
4 years ago @balag0
"ARM API returns the same skiptoken and resources repeatedly when there are no more resources."
I see this comment was yours. Are you sure they were duplicate resources and skip tokens? I have not been able to duplicate that effort.
I really want to get rid of the nextLink depth limits and simply run it to the end because that's the only way to be sure we have retrieved all of the requested resources (subscriptions or otherwise).
nerddtvg
commented
4 years ago @balag0
Have you had a chance to review the above suggestion? I'd love to get your feedback on this so we can get it fixed and usable. I'm hit by this problem almost daily at my job and I'm constantly having to work around it.
balag0
commented
4 years ago Sorry for the delay. The PR is merged now and you can verify if it works for your scenario at https://resources-staging.azure.com/ We can move it to the prod site after a few days if everything looks good.
nerddtvg
commented
4 years ago Thank you! I'll try to confirm tomorrow if time allows.
On Fri, Sep 25, 2020, 9:35 PM Bala G notifications@github.com wrote:
Sorry for the delay. The PR is merged now and you can verify if it works for your scenario at https://resources-staging.azure.com/ We can move it to the prod site after a few days if everything looks good.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projectkudu/AzureResourceExplorer/issues/254#issuecomment-699279812, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJ4QHZYFUFBTD7A5HTX4UTSHVHOXANCNFSM4QFKEYIA .
nerddtvg
commented
4 years ago I can confirm it worked but it took a very long time to load. I don't know if that is because of the efficiency of the code going through that looping, or maybe the resources of the staging environment, but this isn't great. It does work though.
Via https://resources.azure.com/api/providers [10.62 seconds]:
["MICROSOFT.RESOURCES","MICROSOFT.CAPACITY"]Via https://resources-staging.azure.com/api/providers [1.9 minutes]:
["MICROSOFT.COGNITIVESERVICES","MICROSOFT.DATAFACTORY","MICROSOFT.MACHINELEARNING","MICROSOFT.SQL","MICROSOFT.STORAGE","MICROSOFT.WEB","MICROSOFT.ALERTSMANAGEMENT","MICROSOFT.APIMANAGEMENT","MICROSOFT.COMPUTE","MICROSOFT.CONTAINERREGISTRY","MICROSOFT.DBFORMYSQL","MICROSOFT.EVENTGRID","MICROSOFT.INSIGHTS","MICROSOFT.KEYVAULT","MICROSOFT.LOGIC","MICROSOFT.MACHINELEARNINGSERVICES","MICROSOFT.NETWORK","MICROSOFT.PORTAL","SENDGRID.EMAIL","MICROSOFT.RESOURCES","MICROSOFT.CAPACITY"]Via https://resources.azure.com/api/operations/providers/SUBSCRIPTION_ID [9.3 seconds] (expanding a resource group):
{}Via https://resources-staging.azure.com/api/operations/providers/SUBSCRIPTION_ID [1.9 minutes] (expanding a resource group):
{
"RESOURCE_GROUP_NAME": {
"MICROSOFT.ALERTSMANAGEMENT": [
"SMARTDETECTORALERTRULES"
],
"MICROSOFT.APIMANAGEMENT": [
"SERVICE"
],
"MICROSOFT.COGNITIVESERVICES": [
"ACCOUNTS"
],
"MICROSOFT.COMPUTE": [
"AVAILABILITYSETS",
"DISKS",
"VIRTUALMACHINES"
],
"MICROSOFT.CONTAINERREGISTRY": [
"REGISTRIES"
],
"MICROSOFT.DATAFACTORY": [
"FACTORIES"
],
"MICROSOFT.DBFORMYSQL": [
"SERVERS"
],
"MICROSOFT.EVENTGRID": [
"SYSTEMTOPICS"
],
"MICROSOFT.INSIGHTS": [
"COMPONENTS"
],
"MICROSOFT.KEYVAULT": [
"VAULTS"
],
"MICROSOFT.LOGIC": [
"WORKFLOWS"
],
"MICROSOFT.MACHINELEARNING": [
"COMMITMENTPLANS",
"WORKSPACES"
],
"MICROSOFT.MACHINELEARNINGSERVICES": [
"WORKSPACES"
],
"MICROSOFT.NETWORK": [
"NETWORKINTERFACES"
],
"MICROSOFT.PORTAL": [
"DASHBOARDS"
],
"MICROSOFT.SQL": [
"SERVERS"
],
"MICROSOFT.STORAGE": [
"STORAGEACCOUNTS"
],
"MICROSOFT.WEB": [
"CONNECTIONS",
"SERVERFARMS",
"SITES"
],
"SENDGRID.EMAIL": [
"ACCOUNTS"
],
"MICROSOFT.RESOURCES": [
"DEPLOYMENTS"
]
},
"RESOURCE_GROUP_NAME": {
"MICROSOFT.ALERTSMANAGEMENT": [
"SMARTDETECTORALERTRULES"
],
"MICROSOFT.INSIGHTS": [
"COMPONENTS"
],
"MICROSOFT.KEYVAULT": [
"VAULTS"
],
"MICROSOFT.LOGIC": [
"WORKFLOWS"
],
"MICROSOFT.SQL": [
"SERVERS"
],
"MICROSOFT.STORAGE": [
"STORAGEACCOUNTS"
],
"MICROSOFT.WEB": [
"CONNECTIONS",
"SERVERFARMS",
"SITES"
],
"MICROSOFT.RESOURCES": [
"DEPLOYMENTS"
]
}
}
https://github.com/projectkudu/AzureResourceExplorer/blob/df1383227bdbf81f244a62d154bab2cb382043cc/Controllers/ArmRepository.cs#L60-L64
It is possible that requesting resources will return an empty array despite resources existing. You have to use the nextLink multiple times to retrieve them. This seems to be due to a way the ARM API handles RBAC permissions by simply filtering out resources someone doesn't have access to which can return empty or incomplete (fewer than XX entries).
I hope to submit a PR to workaround this issue without giving into the max-depth problem later.
It's possible that because of this weird filtering, what is thought as duplicate skip tokens and resource arrays is actually "correct," just not useful.
This bug hampers the usefulness of Azure Resource Explorer because it doesn't pre-load a lot of providers. When I drill down into a specific resource group, even after clicking "Show All" under providers, I am not allowed to view some that exist. For example, Microsoft.Web should be a resource provider available and it is completely missing. Despite the fact it exists in the full resource list I was able to query from below:
Example using Azure CLI on a subscription where I only have access to a few resource groups: