Workflow classification comparison

[dkmiller]

Compare the following workflow orchestrators:

• Apache Airflow
• Kubeflow
• Azure ML pipelines v1
• Azure ML pipelines v2
• Luigi
• Prefect 1.0
• Prefect 2.0 (Orion)
• Sagemaker
• MLFlow pipelines
• Argo
• Flyte
• Metaflow

Things to consider:

• Supported clouds (AWS, Azure, GCP)
• Static vs. dynamic DAG
• First-class SDKs, language-agnostic REST API
• Pipeline definition \& submission experience
• Extensible?
• Health of community?
• Step / job caching (reuse)
• Recursion / bootstrap?

Existing comparisons:

• https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow

[dkmiller]

https://docs.pachyderm.com/latest/concepts/pipeline-concepts/pipeline/