Closed richox closed 1 month ago
Uniffle is another candidate. Do you mind integrating uniffle with blaze?
Uniffle is another candidate. Do you mind integrating uniffle with blaze?
i suggest we create a abstract layer for integrating different rss implementations. of course we can have one working celeborn/uniffle module, then we can have more idea on designing the abstract layer.
Is your feature request related to a problem? Please describe. apache-celeborn is widely used as an RSS (remote shuffle service) for spark. currently blaze has supported an internal RSS implementation from Kuaishou.Inc, however celeborn is not yet supported.
Describe the solution you'd like integrate blaze with celeborn, this should be done by extending
rss-xxx-repartitioner.rs
. furthermore, we can support an individual rss shim module for supporting different RSS implementations.Describe alternatives you've considered
Additional context