Zen aims to provide the largest scale and the most efficient machine learning platform on top of Spark, including but not limited to logistic regression, latent dirichilet allocation, factorization machines and DNN.
The typical flow of FM/MVM on GraphX is as follows:
val margin = forward(iter)
var (_, rmse, gradient) = backward(margin, iter) {
multi = multiplier(q, iter)
}
gradient = updateGradientSum(gradient, iter)
vertices = updateWeight(gradient, iter)
Logically, margin and multiplier are all temporary variable that all samples in a partition can share them since sample is executed sequentially one by one, thus only one margin/multiplier copy exists. However, in current GraphX implementation, there are many (the number of samples) margin/multiplier variables. This will cost huge memory thus affect the scalability.
The typical flow of FM/MVM on GraphX is as follows:
Logically, margin and multiplier are all temporary variable that all samples in a partition can share them since sample is executed sequentially one by one, thus only one margin/multiplier copy exists. However, in current GraphX implementation, there are many (the number of samples) margin/multiplier variables. This will cost huge memory thus affect the scalability.