Initial Ray runner prototype based on FnApiRunner in Beam

ray-project / ray_beam_runner

Ray-based Apache Beam runner

Apache License 2.0

41 stars 12 forks source link

Closed pabloem closed 2 years ago

pabloem commented 2 years ago

This prototype is hacked together as quickly as possible. I can't speak for its efficiency, and its code cleanliness.

Next steps:

Enable more and more tests under ray_runner_test.py
Make efficiency improvements
- Creation of pcollection holding actors (probably rearchitect)
- Data transfer and encoding
- Improve context passing (serialization and passing-around of RayRunnerExecutionContext, RayBundleContext)

Others...

pabloem commented 2 years ago

fyi @pdames @jjyao @wilsonwang371

wilsonwang371 commented 2 years ago

LGTM. Pablo, can we have a demo on this later sometime?

pabloem commented 2 years ago

thanks Patrick, everyone. I'll try to create new work items for this.

@wilsonwang371 I'll be happy to do a demo on this.