Add docs about error accumulation

[mateuszbaran]

A recent Slack discussion has an excellent example of catastrophic error accumulation on a sphere:

julia> using Manifolds, LinearAlgebra, Manopt, ManifoldDiff, ReverseDiff

julia> N = 10
10

julia> Z = 2*rand(3,N) .- 1
3×10 Matrix{Float64}:
 0.0343791   0.659434     0.881657  …  -0.481577   0.749162  -0.782673
 0.0679203  -0.00865486  -0.532275      0.3281     0.33601    0.799746
 0.151141    0.638862    -0.386737     -0.329479  -0.492241  -0.122683

julia> Z = Z ./ map(norm,  eachslice(Z, dims=2))'
3×10 Matrix{Float64}:
 0.203151   0.718189     0.80144   …  -0.719395   0.782568  -0.695271
 0.40135   -0.00942601  -0.483846      0.490126   0.350993   0.710437
 0.893111   0.695784    -0.35155      -0.492186  -0.51419   -0.108983

julia> M = Sphere(2)
Sphere(2, ℝ)

julia> cost(p) = norm(Z.-p)
cost (generic function with 1 method)

julia> f(M,p) = cost(p)
f (generic function with 1 method)

julia> function grad_f(M,p)
           X = Manifolds.gradient(M,cost,p,Manifolds.RiemannianProjectionBackend(ManifoldDiff.ReverseDiffBackend()))
           println(norm(p), " ", dot(p, X))
           return X
       end
grad_f (generic function with 1 method)

julia> p0 = rand(M)
3-element Vector{Float64}:
  0.48163366734648405
 -0.7087878160523022
 -0.5154113331060776

julia> z = gradient_descent(M,f, grad_f, p0)
0.9999999999999999 4.163336342344337e-16
0.9999999999999999 4.163336342344337e-16
0.9999999999999999 4.163336342344337e-16
0.9999999999999999 4.163336342344337e-16
1.0000000000000002 -9.43689570931383e-16
1.0000000000000002 -9.43689570931383e-16
1.0000000000000002 -9.43689570931383e-16
1.0000000000000004 -1.7208456881689926e-15
1.0000000000000004 -1.7208456881689926e-15
1.0000000000000004 -1.7208456881689926e-15
1.000000000000002 -7.618905506490137e-15
1.000000000000002 -7.618905506490137e-15
1.000000000000002 -7.618905506490137e-15
1.0000000000000129 -4.9758808184918735e-14
1.0000000000000129 -4.9758808184918735e-14
1.0000000000000129 -4.9758808184918735e-14
1.0000000000001046 -4.0033254489202363e-13
1.0000000000001046 -4.0033254489202363e-13
1.0000000000001046 -4.0033254489202363e-13
1.000000000000906 -3.453376473672165e-12
1.000000000000906 -3.453376473672165e-12
1.000000000000906 -3.453376473672165e-12
1.0000000000084301 -3.207698689999816e-11
1.0000000000084301 -3.207698689999816e-11
1.0000000000084301 -3.207698689999816e-11
1.0000000000813305 -3.0918282620555e-10
1.0000000000813305 -3.0918282620555e-10
1.0000000000813305 -3.0918282620555e-10
1.0000000007999785 -3.0396521306930158e-9
1.0000000007999785 -3.0396521306930158e-9
1.0000000007999785 -3.0396521306930158e-9
1.0000000079532962 -3.02114174231035e-8
1.0000000079532962 -3.02114174231035e-8
1.0000000079532962 -3.02114174231035e-8
1.0000000795470854 -3.0211989330251876e-7
1.0000000795470854 -3.0211989330251876e-7
1.0000000795470854 -3.0211989330251876e-7
1.0000007983260109 -3.031773524514751e-6
1.0000007983260109 -3.031773524514751e-6
1.0000007983260109 -3.031773524514751e-6
1.000008027466676 -3.048456460556662e-5
1.000008027466676 -3.048456460556662e-5
1.000008027466676 -3.048456460556662e-5
1.00006733963745 -0.00025574618229285307
1.00006733963745 -0.00025574618229285307
1.00006733963745 -0.00025574618229285307
1.0000673396374502 -0.0002557461822936086
1.0000673396374502 -0.0002557461822936086
1.0000673396374502 -0.0002557461822936086
1.0000673396374502 -0.0002557461822935861
1.0000673396374502 -0.0002557461822935861
1.0000673396374502 -0.0002557461822935861
1.0000673396374502 -0.0002557461822935746
1.0000673396374502 -0.0002557461822935746
1.0000673396374502 -0.0002557461822935746
1.0000673396374502 -0.0002557461822935746
1.0000673396374502 -0.0002557461822935746
[...]

I think this would be a nice thing to document somewhere.

[kellertuer]

Literally a 🚀, since it is leaving earth.

Printing only the first 10 it might be nice to be added to the AD parts in Manopt?

[mateuszbaran]

Sure, it could be documented for Manopt. That's not AD-specific though, it can happen to hand-written gradients as well.