search

novacrazy / dual_num

Dual Number library for Rust
https://docs.rs/dual_num/
17 stars 4 forks source link

Support for nalgebra::MatrixNM for automatic gradient and hessian computation #1

Closed ChristopherRabotin closed 6 years ago

ChristopherRabotin commented 6 years ago

I've fork this thorough implementation of dual numbers in order to add automatic differentiation of vectorial functions. This requires adding a new dependency on nalgebra, and coding a few functions to support hyperdual spaces (and support multiplication of matrices of duals with other such instances). I'll also be adding a thorough test suite.

I was wondering whether you'd like me to open a PR with these breaking changes, or whether to publish this as an independent crate (where I'd mark you as the primary author of course).

I'll be using those changes for https://github.com/ChristopherRabotin/nyx/issues/32.

Let me know what you prefer, I have no preference.

novacrazy commented 6 years ago

That sounds awesome! Although assuming you can leave what already exists intact we can put nalgebra reliant additions behind a feature flag.

Have you committed the new code anywhere yet? To be honest I’m not super well versed in this kind of math, but I’m quite interested in where this could go.

ChristopherRabotin commented 6 years ago

Great! Yes, a feature sounds like the good path forward actually.

I've just pushed some code here: https://github.com/ChristopherRabotin/hyperdual . I'll revert the changes to the Cargo.toml (package name and version). I initially decided to call it hyperdual because the computation of gradients of multivariate functions requires a hyperdual space of the size of the input vector. In practice however, since I didn't quite nail the creation of the hyperspace yet, I'm simulating the hyperspace with N calls to the function by passing each of the independent base vectors. It's kinda gross and expensive, but it works for now.

Do you object to renaming DualNumber to Dual? It's shorter and more related to Real, Complex, and other group friends.

novacrazy commented 6 years ago

Hmm... What would be a better way than iterating over all the dimensions?

For now, if you can put together a branch and/or pull request of the minimum viable changes of just the new additions/tests, I can merge that right away.

Since I saw you star the repo a few days ago I've actually been working on upgrading it to the latest num-traits anyway, so I'll finish that off soon.

I am also in favor of changing DualNumber to Dual, so I'll do that immediately after the merge.

Furthermore, after the PR is merged I would like to add you as a collaborator to the repository if you'd like to make any further improvements. Since you seem to be quite knowledgeable of these things, if you can figure out a good way to do a generic Rem remainder/modulus for dual numbers, I'd be in your debt.

How does that sound?

ChristopherRabotin commented 6 years ago

Sounds good!

I just opened a PR with the basic gradient computation. The PR description lists a few limitations of the implementation, mostly related to the mathematics behind it. I wouldn't want to implement something which isn't mathematically correct.

Note that I didn't do any change related to the update of num-traits. I literally bumped to version 0.2.x and checked that it compiled, and it did.

Concerning your question about not iterating through all the dimensions, I would need to implement a hyperdual space. In practice, line 35 of differentials.rs would change such that the f function isn't called here. This iteration would be used to create a matrix of size N,N (or M, N), and the function f would have to be defined as F: Fn(T, &MatrixMN<Dual<T>, N, N>) -> MatrixMN<Dual<T>, N, N> instead of F: Fn(T, &VectorN<Dual<T>, N>) -> VectorN<Dual<T>, N>. I currently haven't yet implemented that correctly in my demo project where I tested out this math. I'm also not convinced that it would lead to less operations per se since these operations would need to happen on all the items of the matrix anyway. It would look and feel better, but I don't think it would be more computationally efficient.

ChristopherRabotin commented 6 years ago

That was quick! Awesome!

I'm looking forward to work more on this project!

novacrazy commented 6 years ago

I'm currently finishing up stuff with num-traits 0.2, then I'll add you to the repo.

novacrazy commented 6 years ago

I went ahead and made the entire thing based on nalgebra's Vector2. The odds are it'll be used alongside of nalgebra anyway, like in your use case. If that becomes a problem, we can switch back.

Otherwise, your help is appreciated greatly!