search

awslabs / rds-snapshot-tool

The Snapshot Tool for Amazon RDS automates the task of creating manual snapshots, copying them into a different account and a different region, and deleting them after a specified number of days
Apache License 2.0
341 stars 146 forks source link

ListTagsForResources issuing 700 requests per 1 minute #29

Closed kondaps closed 5 years ago

kondaps commented 5 years ago

We are running into API rate limit exceeding errors and per AWS support, the culprit is as below. We need to understand how to correct this issue.

The service team has investigated and has found that the primary API call rate being exceeded is "ListTagsForResource", on 2019-01-09 during a one minute period (at 10:25 UTC), over 700 requests were made, from the following sources:

Boto3/1.7.74 Python/3.6.1 Linux/4.14.88-72.73.amzn1.x86_64 exec-env/AWS_Lambda_python3.6 Botocore/1.10.74
Boto3/1.7.74 Python/3.6.1 Linux/4.14.88-72.73.amzn1.x86_64 exec-env/AWS_Lambda_python3.6 Botocore/1.10.74

Boto3/1.7.74 Python/3.6.1 Linux/4.14.88-72.73.amzn1.x86_64 exec-env/AWS_Lambda_python3.6 Botocore/1.10.74 aws-internal/3 aws-sdk-java/1.11.432 Linux/4.9.124-0.1.ac.198.73.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.192-b12 java/1.8.0_192

The vast majority of these are likely to be the Lambda functions mentioned previously.

mrcoronel commented 5 years ago

Hi @kondaps, could you share a bit more information? 700 seems like an awful lot :) Are you running the latest commit? There's been some improvements on this front How long is your retention period? How often do you backup? How many instances are you backing up? Have you deployed this tool more than once in the same account and region?

kondaps commented 5 years ago

Hi, We tested with latest commit and it did resolve the issue. Thanks a lot for fixing it already :)

Each deployment is backing up a RDS instance and we are noticing these many times calls and failures due to Rate limit reached with old code. Now we no longer has this issue. I believe the same fix is needed for Aurora tool too.