hydro-project / cloudburst

A stateful serverless platform
Apache License 2.0
237 stars 54 forks source link

Adds continuations to Cloudburst DAGs #44

Closed vsreekanti closed 4 years ago

vsreekanti commented 4 years ago

A DAG can be split up into multiple subparts and registered separately. At execution time, you can create a call for each subcomponent using the dry_run flag of the call_dag method and then chain the DAG calls together using the continuation argument of call_dag.

Chained continuations will be sent back to the scheduler, which will then schedule the next portion of the DAG to be executed.