Feature request: automatic differentiation

[alucantonio]

Hi, I have been using gonum to solve minimization problems. I tried using BFGS and things went smooth. I would like to try with Newton's method, so I need to compute the Hessian matrix. Numerical computation of the Hessian is, as expected, very slow and is not an option, in general, for large problems. I can compute the Hessian analytically, but the procedure is error-prone and leads to very complex equations.

I think it would be nice to add an automatic differentiation feature to gonum. This way, the Hessian would be computed automatically and the usability of the Hessian-based minimization algorithms would improve significantly. Go packages implementing this feature exist: autodiff, gorgonia. Do you see potential issues in implementing such a feature, like the need of restructuring gonum minimization routines)? Would it be possible to use one of those packages in gonum?

[chewxy]

You can write one your own: https://github.com/gonum/gonum/tree/master/num/dual

Also, the next version of Gorgonia will feature eager autodiff.

[sbinet]

what gonum could perhaps have is a code generator (based off go/types) to write the derivative code from a given func F(x float64) float64...

[alucantonio]

You can write one your own: https://github.com/gonum/gonum/tree/master/num/dual

Also, the next version of Gorgonia will feature eager autodiff.

Interesting, I didn't know about dual numbers and their application to automatic differentiation. I will take a look at this. Thanks!

[alucantonio]

Ok, I did some preliminary numerical experiments with the num/dual package. In my application I need to differentiate a function of a matrix argument. This generally occurs in Finite Element models for elastic solids, where the function to be minimized is the energy of the system, which depends on the strain tensor. I think that I need to define matrix algebra functions (such as matrix multiplication, trace, norm...) to operate with Number types instead of floats. Do you think it is possible to use the functions of the mat package for this, or I have to generalize/wrap/rewrite them to work with Numbers? Or maybe there is a more direct alternative way (excluding symbolic manipulations to write the energy in terms of scalar components of tensors) I am not considering? What about using the Gorgonia's tensor package to work with tensors based on dual numbers?

[kortschak]

You could use the functions of the mat functions and replicate the operations in the num/dual package. The work there is pretty straightforward.