search

ray-project / ray_beam_runner

Ray-based Apache Beam runner
Apache License 2.0
42 stars 12 forks source link

Let driver own pcollections #41

Closed jjyao closed 2 years ago

jjyao commented 2 years ago

Signed-off-by: Jiajun Yao jeromeyjj@gmail.com

With this PR, driver will own all the data of pcollections and handle the object lose automatically.

Closes #34

pabloem commented 2 years ago

I'll merge after tests pass

pabloem commented 2 years ago

thanks everyone : )

jjyao commented 2 years ago

@iasoon, sorry for the late reply.

I'll apply https://github.com/ray-project/ray/pull/28291 to this PR so we no longer need to calculate num_returns beforehand.