search

rapidsai / spark-examples

[ARCHIVED] Moved to github.com/NVIDIA/spark-xgboost-examples
https://github.com/NVIDIA/spark-xgboost-examples
Apache License 2.0
70 stars 40 forks source link

Run on multiple GPU #64

Closed BlueFelix closed 4 years ago

BlueFelix commented 4 years ago

I use 2 executors and run the xgboost demo. I tried to change spark.sql.files.maxPartitionBytes and maxRowsPerChunk but it always has 1 partition and run on single executor and single GPU. How to use multiple GPU?

BlueFelix commented 4 years ago

It's controled by numWorkers of xgboost's params

anfeng commented 4 years ago

How many data partitions do you have? You need to specify 2 executors and 2 workers.

sameerz commented 4 years ago

Was this issue resolved (numWorkers = number of GPUs)?