search

Azure / PSRule.Rules.Azure

Rules to validate Azure resources and infrastructure as code (IaC) using PSRule.
https://azure.github.io/PSRule.Rules.Azure/
MIT License
394 stars 86 forks source link

Export-AzRuleData may not export all data if throttled #1341

Closed BernieWhite closed 1 year ago

BernieWhite commented 2 years ago

Description of the issue

For high volume subscriptions with many or complex resources, calls to Export-AzRuleData request may get throttled.

Get-AzResource: TooManyRequests : The request is being throttled as the limit has been reached for operation type -
Read_ObservationWindow_00:05:00

Currently PSRule for Azure doesn't specifically handle throttling gracefully and may miss resources.

Expected behaviour

We may need some command line options to configure the behaviour on throttling as timely export of resource may be more important.

However we should provide the ability to export as resources observing throttling window, and take steps to avoid throttling all together if any additional tweaks can be made.

Module in use and version:

Additional reference

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling

ArmaanMcleod commented 2 years ago

Maybe for this we could provide -MaximumRetryCount and -RetryIntervalSec parameters?

I know for the 429 error code, the response does include a Retry-After header indicating how long you should wait until making another request.

abelal83 commented 2 years ago

This is a huge problem when a subscription contains hundreds of resources. There's not much we can do as the call to Azure is handled by Export-AzRuleData.

BernieWhite commented 2 years ago

@abelal83 Thanks for your interest in this issue.