yanboliang / spark-vlbfgs

Vector-free L-BFGS implementation for Spark MLlib
Apache License 2.0
46 stars 17 forks source link

fix VLOR model OOM bug #24

Closed WeichenXu123 closed 7 years ago

WeichenXu123 commented 7 years ago

Now V-LOR use DistributedVector.toLocal to get local coeffs, but the toLocal method will generate the whole dense vector(maybe very large) internally, it will cause OOM problem.

Update the DistributedVector.toLocal method, using sparse vector optimization internally.