CDK Seed/stepfunction event logger

[leothomas]

What I did:

I implemented a CDK construct that takes a list of StepFunction arn's and stores their execution history in dynamodb, or postgres (coming soon) or allows the user to customize their error handling. The event logger is further described here. The event logger

How I did it:

A lot of blood, sweat and tears. Mostly tears because of TypeScript's type checking, that is clearly very powerful, but I have no idea how to properly use.

The seed itself contains a CDK construct that defines:

• 2 SQS Queues (one main, one Dead Letter Queue)
• A datastore (dynamodb for now, postgres coming soon)
• A processing lambda
• All necessary permissions for these resources to talk to each other.

The seed also contains a lambda directory with the code to process the stepfunction execution event

How to test it:

• TODO: unit tests

Check out this very simple example CDK project that contains the CDK seed. Run:

• pipenv install --dev
• cdk deploy stepfunction-logger-test

This will stand up the construct along with a very simple stepfunction that the construct will monitor. Invoke the stepfunction with any parameters for a successful execution, and with {test_failure: true} for a failed execution. Notice in both cases how new events are written to the dynamodb table. It's very easy to query failed events by querying the ExecutionFailed-Timestamp-Index for type = "ExecutionFailed".

To see the construct operating with a "custom" error handling lamdba, set custom_lamdba to True in stack/app.py and re-deploy the stack using the above command. This will re-deploy the stack with a "custom" lambda (in src/lambda) that will instead store events in S3. This is supposed to show how easy it is for the user to supply their own lambda to process the events. This custom lambda can just as easily point to an existing error storage system, fire off emails to responsible people, etc).

TODO:

• [ ] implement CDK infrastructure for postrges
• [ ] implement logic in SQS Message processor lambda to store events in postgres
• [ ] implement unit tests
• [ ] implement configurable + default values for dynamodb table and GSI capacity scaling + arbitrary indexes

[alukach]

I'll wait until you assign reviewers before doing a full review, but I think this construct serves as a great example of how we can write the infrastructure in TS but still use something like Python to buildout the business logic of code. 💯

[leothomas]

@alukach @markdboyd This PR is ready for review, TODO issue is open here