search

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
44 stars 31 forks source link

CORE: SparkUCX memory pool + common RPC classes. #8

Closed petro-rudenko closed 4 years ago

petro-rudenko commented 4 years ago
  1. UcxShuffleConf class - to get SparkUCX configuration.
  2. Memory pool class - memory pool implementation to avoid allocation in the data exchange path. Basically memory pool is a map of size -> buffer list. E.g.: [4kb] -> [buffer1]->[buffer2]->[buffer3]... TODO: LRU cleaning?
  3. Unsafe utils class - to be able to do mmaping files > 2GB and explicitly unmaping rather then relying on java's GC.