Previously, the step calculation incorrectly used xj.copysign(1) instead of 1.copysign(xj). This made the sign determination useless, because signum(1) is always 1. Worse, it multiplied the step with the xj again. This was also reason why the step was zero when xj was zero, hence the check for step being zero. Gradient and Hessian were implemented correctly.
However, there is one problem. When I fixed the step calculation for Jacobian matrix, the scaled Rosenbrock test with the second initial point started to fail for trust region algorithm, because in the first step, the Jacobian was suddenly ill-defined (one column was zero), which caused using LM step instead of Newton-based. Now the real culprit here is probably poor implementation of LM step, but for the time being, we will use always positive step sizes, which hopefully does not harm anyone in practice.
What is also interesting that using larger epsilon also fixed the issue when using negative step size. This shows that we may need some adaptive step size technique to avoid singular Jacobians just because of a small step size.
Previously, the step calculation incorrectly used xj.copysign(1) instead of 1.copysign(xj). This made the sign determination useless, because signum(1) is always 1. Worse, it multiplied the step with the xj again. This was also reason why the step was zero when xj was zero, hence the check for step being zero. Gradient and Hessian were implemented correctly.
However, there is one problem. When I fixed the step calculation for Jacobian matrix, the scaled Rosenbrock test with the second initial point started to fail for trust region algorithm, because in the first step, the Jacobian was suddenly ill-defined (one column was zero), which caused using LM step instead of Newton-based. Now the real culprit here is probably poor implementation of LM step, but for the time being, we will use always positive step sizes, which hopefully does not harm anyone in practice.
What is also interesting that using larger epsilon also fixed the issue when using negative step size. This shows that we may need some adaptive step size technique to avoid singular Jacobians just because of a small step size.