search

Mikolaj / horde-ad

Higher Order Reverse Derivatives Efficiently - Automatic Differentiation library based on the paper "Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation"
BSD 3-Clause "New" or "Revised" License
34 stars 6 forks source link

Delayed-outlined primitive numeric functions #38

Closed Mikolaj closed 2 years ago

Mikolaj commented 2 years ago

Closes #36.

Edit: this is not the most ergonomic design, but it's the least intrusive. For some pain points, once they emerge, I'm sure adequate syntactic sugar or program rewrite tool can be introduced.

Edit2: Future work: add Out variants of several operations at rank S, especially various multiplications of matrices and vectors (for this we need an additional GADT variant of CodeOut for non-rank polymorphic operations spanning ranks and shapes). That should prevent the need to write a lot of Out if everything needs to be delayed. Perhaps write these operations as class methods so that we have the same names for normal and Out versions. This time, the class would not be rank-polymorphic. Operations that vary shapes are going to require not only frontend but also backend changes. Also, add various numeric constraints over Out to the monad constraint set so that the user doesn't need to write them every time when the overlapping instances bug about them.

tomjaguarpaw commented 2 years ago

Interesting. I wasn't expecting it to be done like this. I was expecting "outlining" to be a new mode, alongside forward and reverse.

Mikolaj commented 2 years ago

Interesting. I wasn't expecting it to be done like this. I was expecting "outlining" to be a new mode, alongside forward and reverse.

Hah, that's precisely how I did it for the first half an hour. But I noticed I have to duplicate the delta expression grammars with small changes, replicate variable binding, state, etc., and I thought to myself, what if I just add to what's there instead of copying. The benefit is that delaying can now be done selectively. The drawback is that the old system is slightly polluted with the new mechanism (the extra delta expression constructor for each rank) and that to delay every function that belongs to numeric classes, every non-numeric function application needs to be wrapped in Out, which is verbose for non-numeric portions of the code.

tomjaguarpaw commented 2 years ago

Ah yes, you would have to add a whole new "parallel" grammar of functions.

Mikolaj commented 2 years ago

Thank you for your comments. I've added yet another method of outlining, using laziness. A few other variants would make sense, but this should be enough for experimentation and users can then decide how to develop this further. Merging.

tomjaguarpaw commented 2 years ago

Can we delete this branch?

Mikolaj commented 2 years ago

Yes, it's fully merged, so no point keeping it around. Let me give you admin rights so that you can tidy up as you see fit.

Mikolaj commented 2 years ago

Doh, now I remember I tried and I can't, because we don't have an org, but it's my personal repo. Let me know if I'm missing a way to do that or if you can think of a good org to join/create.