LBFGS-B return wrong result if we choose bad initial value

[yanboliang]

If we choose the initial value which is far from the correct result, LBFGS-B will return wrong result and output error logs. You can refer the following code to reproduce this bug. This function has minimum f(x)=2.0 when x=1.0. But if we choose the initial value far from 1.0, it will output wrong result.

    val solver = new LBFGSB(DenseVector[Double](1E-12), DenseVector[Double](Double.MaxValue))

    val f = new DiffFunction[DenseVector[Double]] {
      override def calculate(x: DenseVector[Double]): (Double, DenseVector[Double]) = {
        val cost = x(0) + 1.0/x(0)
        val grad = DenseVector(1.0 - 1.0/(x(0)*x(0)))
        (cost, grad)
      }
    }

    val nearX0 = DenseVector[Double](1.5)
    val nearRes = solver.minimizeAndReturnState(f, nearX0)
    println(nearRes.x)
    println(nearRes.value)

    val farX0 = DenseVector[Double](1500)
    val farRes = solver.minimizeAndReturnState(f, farX0)
    println(farRes.x)
    println(farRes.value)

The output is:

DenseVector(1.0000013234582712)
2.0000000000017515
ERROR breeze.optimize.LBFGSB - Failure! Resetting history: breeze.optimize.FirstOrderException: Line search failed
ERROR breeze.optimize.LBFGSB - Failure again! Giving up and returning. Maybe the objective is just poorly behaved?
DenseVector(1500.0)
1500.0006666666666

[yanboliang]

@dlwh Do you or other members have plan to solve this issue? We would like to use LBFGS-B in Spark for huber loss regression which hit this bug. Thanks!

[debasish83]

@yanboliang did you try NonlinearMinimizer with projection on bound ? NonlinearMinimizer has ADMM solver as well that supports bound but in my experiments NonlinearMinimizer with project on bound converges faster...Use NonlinearMinimizer.project and pass the proximal for the bound...

import breeze.optimize.proximal.ProjectBox val lb = DenseVector.zeros[Double] val ub = DenseVector.ones[Double] val solver = NonlinearMinimizer.project(ProjectBox(lb, ub))

After that it works like FirstOrderMinimizer...solver.minimizeAndReturnState(huberLoss, init)

I will look into the LBFGS-B error...

[yanboliang]

633 fix this issue, so close it. Thanks!