Quasi-Newton update rules issues

[mateuszbaran]

I'm currently working on some updates to quasi-Newton direction update rules. Here are a few things to check/consider:

• [x] QuasiNewtonCautiousDirectionUpdate doesn't seem to use θ function anywhere.
• [ ] QuasiNewtonLimitedMemoryDirectionUpdate doesn't seem to use scale.
• [x] Some direction updates need a reset mechanism called by initialize_solver!, and maybe also when the direction stops being a descent direction. That shouldn't happen for a well-designed set of options but it's a bit hard to tell which ones are safe from that issue.
• [ ] Unify projections to a custom function instead of a boolean flag.

I'm also working on some variant of direction update for manifolds with corners inspired by L-BFGS-B. There are different approaches with elaborate line searches and update matrices but I'm aiming for something relatively simple and generic. I'm not aiming at competing with Fortran codes, my goal is making combined manifold-box constrained optimization work relatively well.

[kellertuer]

I started looking a bit into these points.

Ad 1) The function is used when updating the Hessian at

https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/solvers/quasi_Newton.jl#L677 or https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/solvers/quasi_Newton.jl#L738

Ad 2) That might indeed be the case, I could only find the scale in the update of the full-matrix variants.

https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/solvers/quasi_Newton.jl#L514-L516

I am not sure where the scaling was “lost” in one of the reworks, but we can surely bring that back

Ad 3) Sure, I am not 100% sure how to realise that, but when we have the iterate available doing a reset when that is zero sounds reasonable. When we wrote this code, reusing the state (and hence the updates) was not so much thought given to, for sure.

• [ ] I also noticed that one of the recent updates that meant to introduce a stability idea did

https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/plans/quasi_newton_plan.jl#L500 and https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/plans/quasi_newton_plan.jl#L582

which we basically have used before but then a bit more generic (not just a boolean) https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/solvers/truncated_conjugate_gradient_descent.jl#L18 and then https://github.com/JuliaManifolds/Manopt.jl/blob/1856a26d8bce827f7fa97176a9cd1b2effbf0863/src/solvers/truncated_conjugate_gradient_descent.jl#L741

which might be a bit nicer to allow also for other ways to care for stability. We should both unify this and probably also switch the second one to embed_project! (since that default is older than the embed/project discussion in Manifolds.jl).

[mateuszbaran]

Ad 1) The function is used when updating the Hessian at

I see, thanks.

Ad 3) Sure, I am not 100% sure how to realise that, but when we have the iterate available doing a reset when that is zero sounds reasonable. When we wrote this code, reusing the state (and hence the updates) was not so much thought given to, for sure.

I have it sketched already so I will only ask for a review when it's ready :slightly_smiling_face:

We should both unify this and probably also switch the second one to embed_project! (since that default is older than the embed/project discussion in Manifolds.jl).

Yes, unifying that is a good idea.

[kellertuer]

If you see how to correctly add the scaling ( and unifying that to be a real value), feel free to add that as well. I am sure it was from one of the Wuang papers about QN.