search

kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
1.27k stars 119 forks source link

Supports apache-celeborn #287

Closed richox closed 1 month ago

richox commented 12 months ago

Is your feature request related to a problem? Please describe. apache-celeborn is widely used as an RSS (remote shuffle service) for spark. currently blaze has supported an internal RSS implementation from Kuaishou.Inc, however celeborn is not yet supported.

Describe the solution you'd like integrate blaze with celeborn, this should be done by extending rss-xxx-repartitioner.rs. furthermore, we can support an individual rss shim module for supporting different RSS implementations.

Describe alternatives you've considered

Additional context

zuston commented 12 months ago

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

richox commented 11 months ago

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

i suggest we create a abstract layer for integrating different rss implementations. of course we can have one working celeborn/uniffle module, then we can have more idea on designing the abstract layer.