search

ray-project / ray_beam_runner

Ray-based Apache Beam runner
Apache License 2.0
41 stars 12 forks source link

Support execution of Splittable DoFn without splitting #14

Closed pabloem closed 2 years ago

pabloem commented 2 years ago

We have extensive tests to verify that we can execute DoFns that split (https://github.com/ray-project/ray_beam_runner/blob/master/ray_beam_runner/portability/ray_runner_test.py#L573-L608) and do SDK-side checkpointing.

We need to make sure this is supported. Please reach out to @pabloem to check what's needed.

pabloem commented 2 years ago

I'll pick up this issue!

pabloem commented 2 years ago

(can't assign to me, but can someone please assign to me?)