kreuzwerker / kreuzlaker

11 stars 2 forks source link

Schedule transformations using stepfunctions #34

Open jankatins opened 1 year ago

jankatins commented 1 year ago

Currently kreuzlaker uses cron lines to schedule tasks, but this might result in problems when tasks depend on each others. Step functions seem an nice solutions for this. Stepfunctions also seem to have a nice solution for notifications on failure.

E.g. see this presentation: https://speakerdeck.com/nicor88/dbt-serverless-how-to-run-dbt-in-your-aws-account?slide=18 -> Stepfunction example