Curious why copy_remote() would fail.

awslabs / rds-snapshot-tool

The Snapshot Tool for Amazon RDS automates the task of creating manual snapshots, copying them into a different account and a different region, and deleting them after a specified number of days

Apache License 2.0

341 stars 146 forks source link

Curious why copy_remote() would fail. #18

Closed jeffmacdonald closed 5 years ago

jeffmacdonald commented 5 years ago

Hi, quite often for me "copy_snapshots_dest_rds" I get failures while running copy_remove() but no visable exceptions. So I looked into the docs of client.copy_db_snapshot() in boto3 and also noticed none.

My backups still work. The eventual retries pick up the pending snapshots and it always works out. But my logs are very noisy and my monitoring system is going a bit berserk :) I could turn down monitoring but I kind of want to know when I don't have backups.

Thoughts?

mrcoronel commented 5 years ago

Hi @jeffmacdonald, RDS will not allow more than 5 concurrent copies across regions. Retries are inevitable, I'm afraid. Maybe you can adjust the Cloudwatch alarms thresholds to account for the amount of retries your particular setup needs? I believe the defaults should allow for two hours before triggering an alarm.

jeffmacdonald commented 5 years ago

Very helpful. Thanks for that!