search

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
44 stars 31 forks source link

Public transport api implementation #25

Closed petro-rudenko closed 4 years ago

petro-rudenko commented 4 years ago

Initial implementation of public transport API for send/recv. Fully working without spark for now.

petro-rudenko commented 4 years ago

@abellina - here's how to run perf tool with GPU:

  1. mvn package (there would be 2 jars: ucx-spark-2.0-for-spark-3.0.0-jar-with-dependencies.jar and ucx-spark-2.0-for-spark-3.0.0-tests.jar
  2. Put them to some jars folder with cudf.
  3. Server: 
    java -cp /PATH_TO_UCX/lib/:spark/jars/*:jars/* org.apache.spark.shuffle.ucx.perf.UcxShuffleTransportPerfTool -i 100 -n 1 -s 5m -m cuda

Client:

java -cp /PATH_TO_UCX/lib/:spark/jars/*:jars/* org.apache.spark.shuffle.ucx.perf.UcxShuffleTransportPerfTool -i 100 -n 1 -s 5m -m cuda server_host:12345
abellina commented 4 years ago

@petro-rudenko should I be reviewing this PR? Let me know if it's not ready.

petro-rudenko commented 4 years ago

@abellina i'll split it (will move benchmark to a separate PR).

petro-rudenko commented 4 years ago

@abellina ready to review

sonarcloud[bot] commented 4 years ago

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

warning The version of Java (1.8.0_265) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more here