Could you explain what advantage is argo-dataflow with regard to flink? flink can be deployed on kubernetes now

[liujiong1982]

Just see this project Could you explain what advantage is argo-dataflow with regard to flink? flink can be deployed on kubernetes now

[alexec]

Thank you for asking this question. Maybe it is a bit like why would you use Argo Workflows as Airflow can be deployed on Kubernetes.

Being “Cloud-native” (i.e. “Kubernetes-native, leveraging its features”) vs “runs on Kubernetes” (i.e “is a Linux process that can run in a container”).

Dataflow uses the Kubernetes APIs, such as leases for HA, CRDs to specify resources, RBAC, for, well, RBAC, the pod API for creating processors, etc etc. If you know Kubernetes, you don’t have to learn new things. You can understand the RBAC, the security policies, and be confident your workloads run safely. You know how resources are allocated to pods, how cluster mange resources And so on. Dataflow and you can therefore focus on the core aspects of data processing.

With dataflow you don’t need to build an image containing your app, and you can use Java, Python, NodeJS or Golang should you wish.

Dataflow is not as mature as Flink, and it’ll be a while before we support a lot of the use cases that Flink supports today, than means Flink will be a better choice for some/many users.

Naturally, we’re not impartial. I’d love to hear from people who’ve used both tools.

[isubasinghe]

Sort of related, but would you consider bringing in https://github.com/timelydataflow/differential-dataflow/blob/master/differentialdataflow.pdf to this project? DD is really nice for performance, fault tolerance and more criteria. I did my thesis on DD and can help here I believe.

[alexec]

I'd love to talk about this, do you want to set-up 30m?

https://bit.ly/book-30m-with-argo-team

[github-actions[bot]]

Stale issue message