search

NVIDIA / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
22 stars 14 forks source link

Project status? #36

Open neverlink opened 5 months ago

neverlink commented 5 months ago

Hi. What is the current state of this project? The Mellanox Repo appears to be legacy. This one, NVIDIA/SparkUCX is a fork of OpenUCX, but this one is ahead by some commits. I find this to be a bit confusing.

From the SPIP Ticket what I understand there is no licensing conflict for the integration in mainstream Spark, but there has been no activity on the ticket since 2020, and no closure to the discussion either.

The Spark UCX website simply times out. :(

From what I've digged up online, this project is a great effort and a huge leap forward for the industry. It appears to be not completely left behind, but it is very unclear whether there is a future for it besides an external plugin.

neverlink commented 1 month ago

Is it that this component lived on to become part of NVIDIA RAPIDS?