yanboliang / spark-vlbfgs

Vector-free L-BFGS implementation for Spark MLlib
Apache License 2.0
46 stars 17 forks source link

Add COO sparse matrix optimization #31

Closed WeichenXu123 closed 7 years ago

WeichenXu123 commented 7 years ago

Add COO storage format for sparse matrix. It will save memory for V-LOR when dimension scales to billions, at the same time won't reduce performance. The reason is that colPtrs array in CSS format matrix will occupy large memory size when the matrix is very sparse.