search

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
44 stars 31 forks source link

[CORE] Reducer protocol. #16

Closed petro-rudenko closed 4 years ago

petro-rudenko commented 4 years ago

Reducer protocol - where all the acceleration happens.

  1. UcxShuffleManager instantiates UcxShuffleReader that instantiates UcxShuffleClient and doing progress.
  2. UcxShuffleClient implements fetchBlocks that's doing 2 consecutive ucp_get operations: 2.1. Fetch offset (OnOffsetsFetchCallback) 2.2. Fetch blocks (OnBlocksFetchCallback)
  3. Removed spark-2.3 support, since this version is not supported by apache anymore.
petro-rudenko commented 4 years ago

Spark-2.3 is unsupported. Nee first merge #12

sonarcloud[bot] commented 4 years ago

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication