search

ray-project / ray_beam_runner

Ray-based Apache Beam runner
Apache License 2.0
42 stars 12 forks source link

Fixes #14: Refactoring ray portable runner #18

Closed pabloem closed 2 years ago

pabloem commented 2 years ago

Making the code more readable and more efficient.

Also adding support for SDF-initiated checkpointing

pabloem commented 2 years ago

r: @iasoon ? : )

fyi @pdames @jjyao @wilsonwang371

pabloem commented 2 years ago

I realize that CI is broken. Should we fix it as part of this PR? or as a follow-up PR?

pabloem commented 2 years ago

I made all the portability runner tests run : )

pabloem commented 2 years ago

TODO: Rename immutable context managers to *Context

pabloem commented 2 years ago

I created https://github.com/ray-project/ray_beam_runner/issues/28 and https://github.com/ray-project/ray_beam_runner/issues/29 to track fixing the comments in the PR. Maybe then we can merge to unblock other reviews? : )

iasoon commented 2 years ago

Sounds good to me :)