Gradient Methods search over parameter space instead of gradient space

[DevonMorris]

Currently in the steepest descent solver has its line search have to implement a line search over parameter space Linesearch<P, F> and all the line search algorithms have their parameter space implement ArgminScaledAdd<P, F, P> and the steepest descent algorithm sets the initial search direction based on the gradient. The combination of these two behaviors essentially enforces that the gradient representation must be identical to the parameter space over which you are optimizing.

This creates limitations when optimizing over a manifold structure which may have a different representation for its parameter space vs its gradient. An example of this may be a lie group

struct RigidBodyTransformation {
     transformation: na::Isometry3<f64>
}

impl ArgminScaledAdd<na::Vector6<f64>, f64, RigidBodyTransformation> for RigidBodyTransformation {
    fn scaled_add(&self, factor: &f64, vec: &na::Vector6<f64>) -> RigidBodyTransformation {
        ... lie maths ...
    }
}

My proposal here is that we perform the line search and the gradient descent over G instead of P. I believe this should be as easy as changing how those traits are defined. From looking at the code, this would entail judiciously substituting a G for P in the LineSearch and LineCondition traits and the corresponding trait bounds in the corresponding solvers.

[DevonMorris]

See PR #354

[DevonMorris]

@stefan-k, this should be closed as a result of my PR. However, I was hoping to use these changes for a project at work. When do you plan to release the next version of argmin?

[stefan-k]

Unfortunately I cannot promise it, but I will try to release a new version by the end of the month. Apologies that currently everything is a bit slower than usual.