Configurable timeout when awaiting snapshot completion on AWS

mikewallace1979 commented 2 months ago

Adds a new config option aws_await_snapshots_timeout and corresponding barman-cloud-backup option --aws-await-snapshots-timeout which can be used to set the amount of time Barman will wait for snapshots to reach a completed state.

This has a default value of 3600 which is significantly higher than the 300 second timeout resulting from the original boto3 default Delay of 15s and MaxAttempts of 40.

Tested by:

Provisioning a new EC2 instance and immediately shutting it down.
Creating three gp2 EBS volumes of 4096GB size and attaching them to the above instance at mount points /dev/sdb, /dev/sdc and /dev/sdd.
Configuring Barman to take snapshot backups of this instance but using a fake ssh_command of ssh barman@localhost so that the volume mount checks are using the local host and not the real instance.
Creating a mock findmnt script and putting it on the path of the barman user - the script is just echo $4 rw,nosuid,nodev,noexec,relatime, so it always appears to find whatever device it is asked about.

This is enough to allow barman backup and barman-cloud-backup to complete a snapshot backup, where the snapshots of the actual volumes in AWS are taken and the new timeout option can be observed to be working.

The backup operation took between 1 and 2 minutes. I tested with aws_await_snapshots_timeout values of below 60s and observed backups to fail, tested with the default value and observed backups to succeed, and tested with values above 120s and observed backups to succeed.

ruimarinho commented 2 months ago

Just hit this issue and would love to see this merged/released. First time snapshots with the disk size I'm using takes much longer than the default 300 seconds, so a configurable timeout would be great!

martinmarques commented 2 months ago

Just hit this issue and would love to see this merged/released. First time snapshots with the disk size I'm using takes much longer than the default 300 seconds, so a configurable timeout would be great!

We are targeting this for the next release. That will be, hopefully, next month

EnterpriseDB / barman

Configurable timeout when awaiting snapshot completion on AWS #961