search

couler-proj / couler

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
https://couler-proj.github.io/couler/index.html
Apache License 2.0
895 stars 86 forks source link

Documentation: Difference to other meta workflow engines #260

Open Make42 opened 1 year ago

Make42 commented 1 year ago

Couler is a meta workflow engine, but I know of at least ZenML and Kedro which are meta workflow engines as well. While the self-presentation of these other two is towards machine learning, it seems to me they are pretty much generally usable for any type of work. What is Couler aiming to do differently than ZenML and/or Kedro?

deepyaman commented 1 year ago

Kedro provides a way to organize your logic (and abstract your datasets, environments, etc.), but it doesn't actually provide a way to deploy to workflow engines like Argo, Airflow, KFP, etc. that run a series of containerized workloads. There are deployment guides and plugins to these, but Kedro could actually integrate with Couler to help deploy Kedro pipelines to various workflow engines (so as to say they're not exclusive or even really overlapping).

Make42 commented 1 year ago

@deepyaman: How about ZenML, which seems to be much closer to Couler in that case? My impression is that ZenML is a bit more machine learning oriented and Couler might be more general-purpose, but fundamentally, they seem pretty similar.

When should I use Couler? Currently, it only supports Argo, so the "being meta" is currently not the case. In contrast, ZenML has already many services to which it can deploy.

deepyaman commented 1 year ago

@deepyaman: How about ZenML, which seems to be much closer to Couler in that case? My impression is that ZenML is a bit more machine learning oriented and Couler might be more general-purpose, but fundamentally, they seem pretty similar.

I think ZenML is much more opinionated about how your workflow gets executed (e.g. you can't directly parametrize the underlying ContainerOps), so it's more about getting your ML workflow to run on a backend as opposed to being able to orchestrate stuff on any backend. That being said, I have superficial knowledge of both ZenML and Couler, and this is based on a cursory read through both codebases. (I am a Kedro maintainer and pretty confident about the first answer.)

When should I use Couler? Currently, it only supports Argo, so the "being meta" is currently not the case. In contrast, ZenML has already many services to which it can deploy.

Agreed that the current value is for deploying to Argo, but I think the whole idea is that the ecosystem can be extended.