search

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
44 stars 31 forks source link

Performance benchmark #27

Open petro-rudenko opened 3 years ago

petro-rudenko commented 3 years ago

To run a benchmark:

mvn package (there would be 2 jars: ucx-spark-2.0-for-spark-3.0.0-jar-with-dependencies.jar and ucx-spark-2.0-for-spark-3.0.0-tests.jar

Put them to some jars folder with cudf. Server:

java -cp /PATH_TO_UCX/lib/:spark/jars/*:jars/* org.apache.spark.shuffle.ucx.perf.UcxShuffleTransportPerfTool -i 100 -n 1 -s 5m -m cuda

Client:

java -cp /PATH_TO_UCX/lib/:spark/jars/*:jars/* org.apache.spark.shuffle.ucx.perf.UcxShuffleTransportPerfTool -i 100 -n 1 -s 5m -m cuda server_host:12345
sonarcloud[bot] commented 3 years ago

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

warning The version of Java (1.8.0_272) you have used to run this analysis is deprecated and we will stop accepting it accepting it soon.Please update to at least Java 11. Read more here