search

civisanalytics / async-graph-data-flow

Asynchronous functions that pass data along a directed acyclic graph
BSD 3-Clause "New" or "Revised" License
16 stars 2 forks source link

Enhancement: Consider Including Edge Conditions #10

Open tolmanam opened 7 months ago

tolmanam commented 7 months ago

I only stumbled across this project by accident, but love what you have done.

You might be interested in looking at some of the things that the Numaflow Project has done for more inspiration.

Numaflow runs jobs in Kubernetes, so the comparisons are only skin deep, but they are creating graphs for processing pipelines and have some nice capabilities that you may be able to implement here.

In particular, I think that the "Edge Conditions" are fairly easy to understand, but incredibly powerful. They allow the users to create routing rules, by encode filter logic into the edges.

I do not have a need for this project right now, but I will certainly keep it in mind and circle back the next time I do.

Thank you for building this.

jacksonlee-civis commented 7 months ago

Thank you for bringing Numaflow to our attention! For async-graph-data-flow, currently the edges are mediated by asyncio.Queue objects, but to your point they can definitely be something much more customizable and powerful. We have experimented a prototype to allow a custom queue as the edges (e.g., apply custom filtering logic, send data to a remote job on the cloud instead of to another asyncio.Task in the same local Python interpreter), but didn't push it to a release because we didn't yet have an actual use case / YAGNI. :-) Thank you again for stopping by and your nice note!