search

turbot / steampipe-mod-aws-thrifty

Are you a Thrifty AWS dev? This mod checks your AWS accounts for unused and under-utilized resources using Powerpipe and Steampipe.
https://hub.powerpipe.io/mods/turbot/aws_thrifty
Apache License 2.0
98 stars 19 forks source link

Cloudwatch Throttle Exception #113

Closed jaredc911 closed 1 year ago

jaredc911 commented 1 year ago

Describe the bug On the check called Unused log streams should be removed if not required, I receive the error: 

operation error CloudWatch Logs: DescribeLogStreams, exceeded maximum number of attempts, 9, https response error StatusCode: 400, RequestID: 8fd06b8d-5f20-403b-a2a3-2ff825760378, api error ThrottlingException: Rate exceeded (SQLSTATE HV000)

Steampipe version (steampipe -v) Example: v0.17.4

Plugin version (steampipe plugin list) aws@0.87.0

To reproduce Using a connection to an AWS account with a lot of Cloudwatch log groups (my account has 320) , click on the dashboard and wait for it to fail with the above error

Expected behavior I expect it to at least return what it has gathered and better yet would be to return what it has and then continue gathering after waiting

Additional context Add any other context about the problem here.

cbruno10 commented 1 year ago

Hey @jaredc911 , sorry you're running into throttling issues!

You can try increasing the max_error_retry_attempts and/or min_error_retry_delay config arg values to see if that will help, as they increase the number of times the plugin retries on retryable errors and how long it initially waits to retry errors (which is then used as a basis when increasing future retry times).

For more info on these config args, please see https://hub.steampipe.io/plugins/turbot/aws#configuration.

Can you please try increasing those values, e.g., max_error_retry_attempts to 15 and min_error_retry_delay to 500, and see if that helps the dashboard load?

jaredc911 commented 1 year ago

Thanks for the suggestion! That seems to have fixed it