search

shinesolutions / aem-aws-stack-provisioner

Puppet provisioner for AEM AWS Stack Builder
Apache License 2.0
4 stars 13 forks source link

Taking a snapshot fails if it taks too long #121

Closed mbloch1986 closed 4 years ago

mbloch1986 commented 5 years ago

In certain circumstances if the live snapshot takes longer than ~15 mins e.g. when there are many changes in a repository (AEM inPlace Upgrade) it will fail and causing the following error message:

[root@ip-10-10-10-10]# /opt/shinesolutions/aem-tools/live-snapshot-backup.sh

/dev/sdb Snapshot Started: Tue Sep 18 14:15:33 AEST 2018

Traceback (most recent call last):

  File "/opt/shinesolutions/aws-tools/snapshot_backup.py", line 295, in <module>

    snapshot.wait_until_completed()

  File "/usr/lib/python2.7/site-packages/boto3/resources/factory.py", line 369, in do_waiter

    waiter(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/boto3/resources/action.py", line 202, in __call__

    response = waiter.wait(**params)

  File "/usr/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait

    Waiter.wait(self, **kwargs)

  File "/usr/lib/python2.7/site-packages/botocore/waiter.py", line 329, in wait

    last_response=response

botocore.exceptions.WaiterError: Waiter SnapshotCompleted failed: Max attempts exceeded
mbloch1986 commented 5 years ago

The wait until completed documentation for snapshot:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Snapshot.wait_until_completed

mbloch1986 commented 5 years ago

At the moment the current time limit for taking a snapshot is 10 minutes. If it takes longer than that the snapshotting will fail. This is due the hardcoded 120 attempts:

https://github.com/mbloch1986/aem-aws-stack-provisioner/blob/master/files/aws-tools/snapshot_backup.py#L248-L253

mbloch1986 commented 4 years ago

Included in release 3.6.0 .