Improve Linesearch and Quasi Newton allocations

[kellertuer]

This PR is work towards having some memory with line searches to reuse candidate (point) memory.

This resolves parts of #333, namely for now

• 6 (in linesearch_ backtrack which now has a variant linesearch_ backtrack!)
• 1, 2, 5 and a mother temporary storage for the direction (formerly \eta_new)

ToDo

• [x] Currently this PR breaks for example CG on the Circle (or Armijo/Wolfe on the circle).
• [x] quasi Newton might now have a copy too less since some side effect now lead to NaN results in the tests – when we have new memory for eta, it works fine for now, but we might want to check where to fix that best otherwise.
• [x] Changelog entry
• [x] Maybe since we are revising step sizes – they are currently very Float64-focussed, but I was not yet able to allow other real types therein

[kellertuer]

Main thing to fix: Circle manifold

Minor thing to fix – check whether we can avoid the eta allocation from (4).

[mateuszbaran]

I've fixed the linesearch error for Circle but not it does not converge for some reason :confused:

[codecov[bot]]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (0bca9b5) 99.65% compared to head (525d4f7) 99.63%.

Files	Patch %	Lines
src/plans/stepsize.jl	88.67%	6 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #335 +/- ## ========================================== - Coverage 99.65% 99.63% -0.02% ========================================== Files 69 69 Lines 6342 6361 +19 ========================================== + Hits 6320 6338 +18 - Misses 22 23 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[mateuszbaran]

I need to fix the failure on 1.6 in Manifolds.jl.

[mateuszbaran]

Done here: https://github.com/JuliaManifolds/Manifolds.jl/pull/692/commits/5fa4eed08f6abd8e73599b7cc0d1f8dac53cb740 .

[kellertuer]

Okay then let's wait for that one.

We are missing code coverage here anyways probably; and I would like to check the docs the next few days calmly again.

[kellertuer]

Oh and you found a way for the qN update – super neat!

[mateuszbaran]

It was just a minor bug :slightly_smiling_face:

[kellertuer]

Now just test coverage is missing for the (old, allocating) Quasi Newton Updates. And of course if we see a good way to remove the 10 missing lines from WolfePowell that would be great, it is one of the few places where I was not yet able to hit that case in tests.

[kellertuer]

I got the overall coverage to be fine again; but the patch one is not fulfilled since we now have a few more lines in the uncovered case in WolfePowell

[kellertuer]

@mateuszbaran can you check the fail on 1.6? I though we fixed that for circle with the PR for Manifolds.jl 0.9.9...

[mateuszbaran]

We've fixed another, similar issue but not this one... I will fix this.

[kellertuer]

Oh, then I confused that, but thanks for fixing this so fast again :)

[kellertuer]

Hm, I am not aware we did breaking changes in Manifolds is ManifoldsBase but it seems the same thing you just fixed on Circle also happens on Euclidean.

[mateuszbaran]

IIRC it's not breaking, it's just a bug that was exposed by these changes. I will fix that too.

[mateuszbaran]

(exposed by changes in this PR)

[kellertuer]

Thanks!

Suer fixing bugs is nice, and they all seem to be the same area, but it is annoying to check for ;)

[mateuszbaran]

It's a really weird case I'd probably dismiss with "don't do that" if it didn't come up as a test failure here :sweat_smile:

[kellertuer]

The increase in code coverage is unfortunate but the one uncovered block this project has is now a few lines longer. So this should still be fine.