ArgoLabs Dataflow is on hold - use Numaflow

argoproj-labs / old-argo-dataflow

Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.

Apache License 2.0

269 stars 31 forks source link

ArgoLabs Dataflow is on hold - use Numaflow #529

Closed alexec closed 1 year ago

alexec commented 2 years ago

~We've put Dataflow on hold as we are planning on some significant changes and redesign. Stay tuned.~

Dataflow has been rewritten and is now available as Numaflow

Bouwdie commented 2 years ago

@alexec we would love to start testing dataflow, do you maybe have some insights in what the changes will be? and how fundamental :dancers:

alexec commented 2 years ago

We're currently working a new version:

It's focussed on providing ways to do windowing operations. This is something difficult for the old version.
It's currently supports Kafka, does not provide support for the variety of source and sinks.
It has many similar concepts and ideas as this version (e.g. pipelines are CRDs).
It should be an easy migration from the older version to this version.
It should be available in about 3mo.

With that in mind, any testing you do with old dataflow, will be extremely instructive and helpful.

cdeboeser commented 2 years ago

Already excited to see this unfold! Are you looking for contributors?

Also another question: Is there any aspiration to allow for batch jobs, ie reading a bounded source including fan-out and -in? Use case is finite data processing jobs, along ETL lines really.

ant31 commented 2 years ago

Hello, Why is it not built in the opensource, isn't argo a CNCF project?

BenCoughlan15 commented 2 years ago

Hey, is this still in the works? I'm wondering if Nats streaming will still be supported?

dafang commented 2 years ago

Hey, is this still in the works? When this can be used in production?

openrory commented 2 years ago

Hey guys, any update on the re-launch date yet?

louisnow commented 2 years ago

Argo 3.4 rc1 has removed dataflow pipelines from the codebase

https://github.com/argoproj/argo-workflows/issues/8998

From earlier in the thread, the idea was that it should be possible to upgrade easily .

It should be an easy migration from the older version to this version.

Will that still be the case @alexec

alexec commented 2 years ago

@edlee2121 can I please ask you to comment on this issue to give users insight into the future.

edlee2121 commented 2 years ago

Hi all, dataflow is being reworked as part of a new project focused on operational analytics. Please check it out. https://github.com/numaproj/numaflow

vigith commented 1 year ago

Now that we have open-sourced numaflow - https://github.com/numaproj/numaflow, shall we close this issue?