Add stabilization to AWS::StepFunctions::Activity resource's Delete handler
Motivation:
DescribeActivity operation is eventually consistent. The results are best effort and may not reflect very recent updates and changes. This results in the Read handler not always retrieving the correct state after the deletion of an activity. (e.g. Invoke Delete handler, and the immediately invoking Read handler may return the activity details despite the activity having been deleted)
This change follows CloudFormation's best practices in how to develop resource handlers:
DO NOT delegate the response of the model, check existence, or stabilization in CREATE, UPDATE and DELETE handler to the READ handler
This change adds the following logic to stabilize the deletion:
After calling DeleteActivity, we now call DescribeActivity consecutively for STABILIZATION_CALL_COUNT = 10 number of times.
If all DescribeActivity calls fail, then the Delete handler returns SUCCESS
If any DescribeActivity successfully returns the deleted activity, we set callBackDelaySeconds based on a rudimentary ExponentialBackoffDelayWithJitter helper method, and return IN_PROGRESS. This will allow the Delete handler to retry the 10 consecutive DescribeActivity calls after callBackDelaySeconds has passed.
We allow 3 retry attempts, and if all attempts fail, we return SUCCESS to preserve backwards compatibility.
The above logic is made possible by setting variables in the CallbackContext, so each invocation of the Delete handler (and callbacks) has context on whether the handler has already deleted the activity/which retry attempt we are on.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available: N/A
Description of changes:
Add stabilization to
AWS::StepFunctions::Activityresource's Delete handlerMotivation:
DescribeActivityoperation is eventually consistent. The results are best effort and may not reflect very recent updates and changes. This results in the Read handler not always retrieving the correct state after the deletion of an activity. (e.g. Invoke Delete handler, and the immediately invoking Read handler may return the activity details despite the activity having been deleted)This change follows CloudFormation's best practices in how to develop resource handlers:
This change adds the following logic to stabilize the deletion:
DeleteActivity, we now callDescribeActivityconsecutively forSTABILIZATION_CALL_COUNT = 10number of times.DescribeActivitycalls fail, then the Delete handler returnsSUCCESSDescribeActivitysuccessfully returns the deleted activity, we setcallBackDelaySecondsbased on a rudimentaryExponentialBackoffDelayWithJitterhelper method, and returnIN_PROGRESS. This will allow the Delete handler to retry the 10 consecutiveDescribeActivitycalls aftercallBackDelaySecondshas passed.SUCCESSto preserve backwards compatibility.The above logic is made possible by setting variables in the CallbackContext, so each invocation of the Delete handler (and callbacks) has context on whether the handler has already deleted the activity/which retry attempt we are on.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.