search

szilard / GBM-multicore

GBM multicore scaling: h2o, xgboost and lightgbm on multicore and multi-socket systems
20 stars 1 forks source link

H2O can be NUMA-aware if the JVM is run with numactl #2

Open Laurae2 opened 7 years ago

Laurae2 commented 7 years ago

cf https://0xdata.atlassian.net/browse/PUBDEV-3625

4 JVMs NUMA-aware (128 threads each): 14 minutes
for i in 0 1 2 3; do
    sudo nohup numactl -N $i -m $i \
    java -Xmx256G -jar h2o-3.10.0.9/h2o.jar 2>h2o.err 1>h2o.out &
done
1 JVM: 19m 40s
java -Xmx1024G -jar h2o-3.10.0.9/h2o.jar 2>h2o.err 1>h2o.out

1 JVM: 6m
4 JVMs: 30m+
4 JVMs NUMA-aware: 23m40s