search

h2oai / h2o4gpu

H2Oai GPU Edition
Apache License 2.0
460 stars 95 forks source link

Optimize memory usage of elastic net for multiple gpus #763

Closed sh1ng closed 5 years ago

sh1ng commented 5 years ago

Original request from a customer #762

Elastic net fails when tries to allocate memory on a first GPU while a few are available.
https://github.com/h2oai/h2o4gpu/blob/master/src/gpu/matrix/matrix_dense.cu#L1945-L2016

To consider smarter usage of upload_data as it accepts device as a parameter.

sh1ng commented 5 years ago

Example notebook Untitled (1).zip

hemenkapadia commented 5 years ago

Hi @sh1ng, is there a time frame to get this resolved ?

sh1ng commented 5 years ago

I hope to start in a week or two.

hemenkapadia commented 5 years ago

Hi @sh1ng , checking if you were able to make progress on this one. There is an open customer support ticket for it and we would like to get it resolved sooner.

sh1ng commented 5 years ago

I'm to work on it next week.

sh1ng commented 5 years ago

My original analysis was wrong.
With current implementation every GPU has to have it's own copy of a whole data-set to utilize data parallelism, i.e. every GPU processes part of alpha/lambda parameters, but algorithm itself is not parallelised to multiple GPUs.

Need a bit of more analysis to understand how(if) feasible it is. cc @pseudotensor

sh1ng commented 5 years ago

We decided to use xgboost for linear models. At this point it scales enough and I'm able to handle data set that fails with ElasticNet.

snippet

param = {'objective': 'reg:squarederror',
         'booster': 'gblinear',
         'updater': 'gpu_coord_descent',
         'n_gpus': -1,
         }

dtrain = xgb.DMatrix(x, label=y)

parameter gpu_coord_descent is not documented yet.