search

ray-project / ray_beam_runner

Ray-based Apache Beam runner
Apache License 2.0
41 stars 12 forks source link

Add watermark-based scheduling to the Ray Runner #15

Open pabloem opened 2 years ago

pabloem commented 2 years ago

The Ray Runner currently works by topologically sorting the pipeline graph, and executing stage by stage until the whole pipeline has been executed. This means that it only supports batch mode, and it can't execute multiple stages in parallel.

By implementing watermark-based scheduling, and by executing any bundle that is ready for execution, we can start gaining parallelism, and move towards streaming support.

This work is somewhat involved, because it requires changing the whole execution logic for the pipeline, however it should increase our parallelism, which will be great (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py#L420-L487)

pabloem commented 2 years ago

Useful literature: https://s.apache.org/beam-fn-api

wilsonwang371 commented 2 years ago

what is our plan to add watermark-based scheduling? any reference implementation that we need to discuss? for example: flink?

rkenmi commented 1 year ago

Hi @pabloem @iasoon , is this PR https://github.com/ray-project/ray_beam_runner/pull/24 associated to this issue? I might be interested in picking this up.