Added results from Giles 2.2-2.3

[adamhaber]

Added matrix arithmetic results following Giles 2.2.1-2.3.3. Decided not to add 2.3.4/5 (Matrix polynomial/exponential) at this point because they seem to require more verbose explanation, which doesn't really fit the current format. Easy to change if you think differently. Also, decided to only include tangents and adjoints (and not derivatives) to be consistent with Giles.

If this seems alright I can continue to sections 3-5 (linear algebra).

[bob-carpenter]

I think the question is how verbose to make these explanations. Giles goes through derivations. The simple univariate ones don't need any explanation, but I'd think that some of these matrix ones could use some more explanation.

@charlesm93 wrote the matrix exp for Stan, so he'll probably have an opinion on matrix exponential.

[bob-carpenter]

@adamhaber Personally, I find it helpful to have the basic derivatives as well as the forward and backward rules. What do you think?

[adamhaber]

I also think it's helpful. However, for anything past addition/multiplication, these derivatives are not on Giles, and I'm not comfortable with doing this myself without a reference (especially for the various decompositions).

Do you want me to add just the simple derivatives, and leave the rest as it is?

[bob-carpenter]

Do you want me to add just the simple derivatives, and leave the rest as it is?

Sure.

for anything past addition/multiplication, these derivatives are not on Giles, and I'm not comfortable with doing this myself without a reference

I'd suggest these two:

• https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

• Magnus and Neudecker's Matrix Differential Calculus.

Magnus and Neudecker used to be available for free from one of the authors' web sites, but now the 3rd edition's been officially published and it looks like they took it down. It's a great book on matrix calc.

[charlesm93]

@charlesm93 wrote the matrix exp for Stan, so he'll probably have an opinion on matrix exponential.

In Stan, I simply used the direct method, which if I recall correctly is what ends up being recommended in Giles. Not sure what folks do for the action of the matrix exp. I recasted it as an ODE solution and then worked out the adjoint ODE system which is solvable with a matrix exponential. I need to find the paper where I worked this out (or do it again).

[adamhaber]

@bob-carpenter I've added derivatives for the simple cases, and I'll search the refs for the other derivatives.

@charlesm93 Direct method = for loop that accumulates the contributions of the different terms of the polynomial?

[charlesm93]

@adamhaber Re: direct method. Yes, that's right. So you autodiff the approximation, which is brute-force.