sandys
commented
3 years ago Thank you for taking this forward.
We are also seriously looking at Ray Distributed - because of its native Integration with kubernetes ( https://docs.ray.io/en/master/cluster/kubernetes.html ) and its success at Ant Financial in China ( https://docs.ray.io/en/master/cluster/kubernetes.html )
My thought is that you won't be able to make it generic enough to fit all frameworks. And probably it's an overkill of engineering to do so.
The one thing I would strongly request you to align towards is kubernetes compatibility. Because it's the management framework that has massive amount of support.
It is for this reason that Torch chose to write a thin wrapper over kubernetes for its distributed training library - https://pytorch.org/elastic/0.2.1/index.html
So I would suggest when you look at solving this problem from a kubernetes first standpoint. Compatibility with every framework is going to take up more of your time than you might be willing to spend.
Sincerely Sandeep
On Thu, 17 Dec, 2020, 19:00 Jiaming Yuan, notifications@github.com wrote:
Right now, integration between dask and various single node machine learning libraries are implemented as standalone dask extensions like dask-ml and dask-optuna. These can be used with xgboost when xgboost is performing single node training. That's using XGBRegressor and friends with them, instead of using xgboost.dask.DaskXGBRegressor. If users want to train the entire dataset on 1 model, the dask interface is required. The underlying issue is xgboost by itself is a distributed learning library employing a MPI like communication framework, but those extensions are designed to extend single node libraries. To resolve it, we need to design python wrappers that can glue them together.
Optuna is an exception as it's using callback function in xgboost, so the xgboost.dask interface can be adopted to optuna. I will submit some changes with demos later. Others like grid searching are more difficult to implement.
Related: #5347 https://github.com/dmlc/xgboost/issues/5347
cc @pseudotensor https://github.com/pseudotensor @sandys https://github.com/sandys
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmlc/xgboost/issues/6525, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYU3WUDQSFYIEA2IBT6DSVIBVRANCNFSM4U7VFG7Q .
trivialfis
CarterFendley
Right now, integration between dask and various single node machine learning libraries are implemented as standalone dask extensions like dask-ml and dask-optuna. These can be used with xgboost when xgboost is performing single node training. That's using
XGBRegressorand friends with them, instead of usingxgboost.dask.DaskXGBRegressor. If users want to train the entire dataset on 1 model, the dask interface is required. The underlying issue is xgboost by itself is a distributed learning library employing a MPI like communication framework, but those extensions are designed to extend single node libraries. To resolve it, we need to design python wrappers that can glue them together.Optuna is an exception as it's using callback function in xgboost, so the xgboost.dask interface can be adopted to optuna. I will submit some changes with demos later. Others like grid searching are more difficult to implement.
Related: https://github.com/dmlc/xgboost/issues/5347
cc @pseudotensor @sandys