search

awslabs / aws-solutions-constructs

The AWS Solutions Constructs Library is an open-source extension of the AWS Cloud Development Kit (AWS CDK) that provides multi-service, well-architected patterns for quickly defining solutions
https://docs.aws.amazon.com/solutions/latest/constructs/
Apache License 2.0
1.22k stars 247 forks source link

New feature: aws-stepfunctionstasks-sqs #280

Open knihit opened 3 years ago

knihit commented 3 years ago

Step function has a callback pattern called 'Callback with Task Token' - https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html. It allows for slower back-ends to respond rather than getting throttled and fail.

Use Case

Most back-ends and AWS services have TPS limits may not be able to handle bursts or higher volume of streaming data. Having step function workflow, retries and data streams like MSK or Kinesis does reduce the throttling issues. But does not solve the problem where a step function task has a slower back-end and a successful completion of a task requires transitioning through all the states in the statemachine.

Proposed Solution

To use the callback pattern, stepfunctions can have task that is backed with a queue (like SQS). On the consumer side of the queue, the target resource (Ex: lambda, a containerized implementation, or an EC2 instance, processes the message and sends the status back to the state. This triggers the state machine to transition to the next step.

The aws-stepfunctions-sqs (a statemachine task that is backed by SQS) can be combined with the existing aws-sqs-lambda construct to complete this pattern (lambda specific version of the stepfunction callback pattern). Note: additional permissions are required to be assigned, in this case to the lambda function to be able to send back the task status to the statemachine.

As part of this construct, the proposal is to create a state machine fragment (https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html) that would allow it to be chained to a state machine definition.

Other

https://zaccharles.medium.com/async-callbacks-with-aws-step-functions-and-task-tokens-9df97fe8973c https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions.StateMachineFragment.html

Readme

here are more details about the actual construct with its README and architecture diagram.

This is a :rocket: Feature Request

knihit commented 3 years ago

Renaming it to aws-stepfunctionstasks-sqs, since it is the tasks which are backed by SQS not the stepfunctions itself. Also to make it composable with a larger statemachine definition, the construct should create a StateMachineFragment

knihit commented 3 years ago

@biffgaut, here are more details about the actual construct with its README and architecture diagram.