nccl as a submodule, xgboost live-lock fix

h2oai / h2o4gpu

H2Oai GPU Edition

Apache License 2.0

460 stars 95 forks source link

nccl as a submodule, xgboost live-lock fix #721

Closed sh1ng closed 5 years ago

sh1ng commented 5 years ago

Updated NCCL to 2.4.2
No need to install NCCL separately
It works on pcc64le with CUDA8/9
~~xgboost's live-lock still occurs(I hoped it fixed in a new version). Looks like it's xgboost issue https://github.com/dmlc/xgboost/issues/4107~~ Merged a temporary fix. Deadlock can still occur on CUDA 8(know issue of NCCL)
Added workaround(cache build in docker cache) to not compile NCCL every time. Another option could be to store it in s3, but that introduces a dependency on H2O infrastructure.

trivialfis commented 5 years ago

@sh1ng Latest NCCL doesn't solve the problem, I just tried to compile NCCL from source. I will workaround the problem by using old Group* APIs of NCCL in XGBoost.