Derivatives for mutating linear algebra

[ChrisRackauckas]

Things like mul! are important here, given the DualCache style of support. @oxinabox how should we be implementing derivatives for these? @jrevels mentioned that ChainRules.jl does have support for it. We need to basically say "check to make sure it's a dual of the right size, and then if so, use these mutating functions ..."

[oxinabox]

I have zero ideas for how ChainRules can be enhanced to support them. We need to sit down with @willtebbutt and @mikeinnes and think hard.

[shashi]

I might be stating the obvious because I'm just starting to think about this, I think the only way to do this is to use the fact that Dual is not user facing API. But we're allowed to do any program transform inside the dualrun context...

So some obvious things might get us going:

• if the program has a x[i] = a then it can start a dual number cache for x: we can also make this thing tightly typed using errors and restart (I wish every day that we had continuations), but if we hope that, for a setindex, the type of the dual does not change (I bet it does not change more than a couple of times in any good use case) then we're in good shape.
• Of course a vector ChainRule + DualArray might come in and make this very fast (only the transformed program sees the DualArray, but the "primal code" (I'm going to call it that from now on) just sees the primal array. (@YingboMa For this to work, I think we should make primals in DualArray a separate array (which is straight up what the primal code gives you) to support the user's assumptions about it. Duals can still be a n+1 dimensional array of whatever type is going to be fast.)
• For falling back to scalar rules, x[j] can be hijacked to look up this cache for its dual parts.