search

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
44 stars 31 forks source link

[CORE] Mapper protocol implementation. #15

Closed petro-rudenko closed 4 years ago

petro-rudenko commented 4 years ago

Mapper protocol implementation. A main entry point is class UcxShuffleBlockResolver that overrides writeIndexFileAndCommit method. In this method, it registers data and index files and publishes metadata information to driver.