mcabbott
commented
3 years ago There's another example here: https://github.com/TuringLang/DistributionsAD.jl/pull/23#issuecomment-575671303. Behaviour on tagged ForwardDiff:
using ForwardDiff, LinearAlgebra
A = Matrix(I, 2,2)
ForwardDiff.gradient(x -> sum(x \ [1,3]), A) # [-1 0; 0 -3]
ForwardDiff.gradient(x -> sum(cholesky(x) \ [1,3]), A) # [-1 -4; 0 -3]
Aplus2 = [1 0; 0.001 1] # perturb A[2]
sum(Aplus2 \ [1,3]) - sum(A \ [1,3]) # -0.001
sum(cholesky(Aplus2) \ [1,3]) - sum(cholesky(A) \ [1,3]) # PosDefException: matrix is not Hermitian
Aplus23 = [1 0.001; 0.001 1] # perturb A[2] and A[3] together
sum(Aplus23 \ [1,3]) - sum(A \ [1,3]) # -0.004
sum(cholesky(Aplus23) \ [1,3]) - sum(cholesky(A) \ [1,3]) # -0.004Perhaps one could claim the cholesky result is particular sub-gradient. But will the algorithm always behave this way? I'm don't know. The one for simple \ is silently wrong --- or at least, it has promoted A to be structurally diagonal. The Jacobians are:
ForwardDiff.jacobian(x -> x \ [1,3], A)
# -1.0 -0.0 -0.0 -0.0
# -0.0 -0.0 -0.0 -3.0
ForwardDiff.jacobian(x -> cholesky(x) \ [1,3], A)
# -1.0 0.0 -3.0 0.0
# 0.0 0.0 -1.0 -3.0With the proposed change to ==, the results instead match finite differences. Here \ may choose a different algorithm, but this shouldn't cause a discontinuity of the forward pass.
ForwardDiff.gradient(x -> sum(x \ [1,3]), A) # [-1 -3; -1 -3]
ForwardDiff.gradient(x -> sum(cholesky(x) \ [1,3]), A) # ERROR: PosDefException
ForwardDiff.jacobian(x -> x \ [1,3], A)
# -1.0 0.0 -3.0 0.0
# 0.0 -1.0 0.0 -3.0
Aplus2 = [1 0; 0.001 1]
Aplus2 \ [1, 3] - A \ [1, 3] # [0, -0.001]
Aplus3 = [1 0.001; 0 1]
Aplus3 \ [1, 3] - A \ [1, 3] # [-0.003, 0]The example just above that, in https://github.com/TuringLang/DistributionsAD.jl/pull/23#issuecomment-574965028, also gives an error on this branch, also from cholesky. As does a finite perturbation.
mateuszbaran
ChrisRackauckas
devmotion
bvdmitri
Consider this function:
Here
ForwardDiffgets the wrong answer, according to your first calculus class: The derivative is defined by taking limits, evaluatingsq(x + ε)for some smallε, and these always see the continuumx^2not the special point.One to think about this is to say that
x==1really meansabs(x-1) < ζwith some tinyζ, which we keep finite until we are sure we aren't confused. The calculus class assumption is thatζ << ε.The assumption of
ForwardDiffis the opposite. ItsDual(x,1)encodes a perturbationx + 1εwithεsmaller than everything else around, and in particularε << ζ. Or in other words,sqis viewed as being piecewise continuous, with a small flat area of width2ζ, which is still large enough for us to see that its slope is zero.Of course nobody really writes contrived examples like
sq. But they do write things like this:This has almost the same problem as #197, where
det(A)tests foristriu(A) || istril(A)before calling a simpler branch. The fact thatf(x,y) == g(x,y)wheny==0does not imply thatdf/dy == dg/dy. So it seems AD ought not to take that branch.In which case we want something like this:
This fixes the tests above, and (a slightly more careful version) fixes #197 and #407.
However, it means that
fun(Dual(x,1)).valueneed not be equal tofun(x), on a discontinuous function. Althoughfun(Dual(x,0)).valueshould still be equal,@assert zero(x) == 0isn't broken, and there should be no problems where functions use things likezero(eltype(xs))for type-stability.The idea that the forward evaluation is unchanged is often thought of as an axiom of AD, but for discontinuous functions, I think that's another way of saying
ε << ζ. Which is a choice. And one that your calculus teacher would disapprove of. The point of evaluating a function with dual numbers is, presumably, to find derivatives, so finding them correctly ought to have a higher priority.There are other comparisons to think about, for example:
I'm not sure how often simulating
x==1as insq2(x)happens in the wild. Perhaps from some combination likef(x) = relu(x) + 0.1*relu(-x)?But
clamping parameters to some range is routine. Your calculus teacher would throw an error here, but that's probably not the most helpful response for the computer.Returning a nonzero derivative here is useful because, if this is some parameter being optimised, it means gradient descent won't get stuck against the wall, when the gradient is away from it. So you can argue that the ability to choose which sub-gradient
ForwardDiffwill use is a feature. The0.5gradient allaFiniteDifferenceswould also be fine for getting un-stuck, but it's very difficult to picture howForwardDiffcould evaluate both branches, and easy to picture doing so having awful side-effects.Here is one way to relate the present rule for
>(::Dual, ::Real)and>=(::Dual, ::Real)to the finite-everythingζ << εstory. We can say that while theε-ball overlaps with both sides, the vote from the longer side (longer by about2ζ) always wins by a hair:Trying out the above
==(::Dual, ::Real)rule, it looks like the tests of this package all pass, except for the ones explicitly testing such rules. It would be interesting to know if this breaks any other uses in the wild. It would also be interesting to think up other pathological examples, maybe I've missed something important.Also:
Another way to talk about this: The problem with
prod2above, anddetin #197, is that they promote accidental zeros of the input to structural zeros. And AD then respects these, and gets the wrong answer. What looked like a simple optimisation when writing for real numbers, has been unintentionally promoted to a constraint on what derivatives are allowed. This is the reverse of the discussion about preserving structural zeros in things likeZygote.gradient(sum∘exp, Diagonal(ones(3)))[1].Some other packages get this right, such as TaylorSeries.jl, this C++ code, and this Ruby. Some get it wrong (according to me) like this Mathematica code, this paper with Matlab, and this blog post, although he changed his mind from right to wrong. More mathematical treatments seem to regard the tuple
(x,δx)as inheriting==from tuples, i.e. they get it right.Similar things were also discussed in #377, where the example is this:
[ForwardDiff.derivative(f, 0.0) for f in [sq, sq2, sq3, sq4, sq5]] == [0,0,0,0,0] [ForwardDiff.hessian(x -> f(x[1]), [0.0])[1] for f in [sq, sq2, sq3, sq4, sq5]] == [2,2,-2,2,2]