Currently, in each V-LOR iteration, the LBFGS iteration states depends on the last 10 states, and LBFGS history caching last 10 coeffs and grads.
Add checkpoint at iterationNumber % iterationInterval == 0, checkpoint the status, containing coeffs, grad and adjustedGrad(used in OWLQN). I do not checkpoint all of last 10 states because such way already truncate the RDD lineage at the checkpoint place for the future states generated after the next 10 states.
Add checkpoint for VLogisticRegression.
Currently, in each V-LOR iteration, the LBFGS iteration states depends on the last 10 states, and LBFGS history caching last 10 coeffs and grads. Add checkpoint at iterationNumber % iterationInterval == 0, checkpoint the status, containing coeffs, grad and adjustedGrad(used in OWLQN). I do not checkpoint all of last 10 states because such way already truncate the RDD lineage at the checkpoint place for the future states generated after the next 10 states.