search

ray-project / ray_beam_runner

Ray-based Apache Beam runner
Apache License 2.0
41 stars 12 forks source link

Add persistence of side inputs into state after they are calculated / executed #13

Closed pabloem closed 1 year ago

pabloem commented 2 years ago

Side inputs are not supported at the moment. To test support, we only need to enable this test:

https://github.com/ray-project/ray_beam_runner/blob/master/ray_beam_runner/portability/ray_runner_test.py#L151-L163

The code exists to support this (https://github.com/ray-project/ray_beam_runner/blob/master/ray_beam_runner/portability/ray_fn_runner.py#L455-L459), but we need to figure out how the protocol to write and access side input state.

We need first to support state (issue #12 ) and then implement this.

pabloem commented 2 years ago

Useful literature: https://s.apache.org/beam-fn-api

pabloem commented 1 year ago

this was implemented in https://github.com/ray-project/ray_beam_runner/pull/44 with @iasoon implementation plus some fixes to data races