search

JuliaDiff / Capstan.jl

A Cassette-based automatic differentiation package for the Julia language
Other
56 stars 7 forks source link

Ideas from "First-Class Automatic Differentiation in Swift: A Manifesto" #4

Open datnamer opened 6 years ago

datnamer commented 6 years ago

https://gist.github.com/rxwei/30ba75ce092ab3b0dce4bde1fc2c9f1d

@jrevels this is a very very interesting read that has some potential ideas for Julia's AD ecosystem.

@jekbradbury knows more.

jekbradbury commented 6 years ago

There was lots of discussion of this and related things in the #autodiff channel today; perhaps it’s worth moving some of that to a more permanent place

jrevels commented 6 years ago

Yes, I have read it. Copy/pasting/editing some of my comments from Slack:

This manifesto aligns fairly well with my direction. I believe that Capstan can probably be a bit more ambitious in the long run, though, since Julia can exploit runtime type info in ways Swift currently cannot.

It seems like the Swift team will soon have the machinery in place to develop mixed-mode, but are not necessarily concerned with mixed-mode as a first-class goal right now. I think this is unfortunate, as a mixed-mode approach can loosen restrictions on target language programmability without sacrificing as much performance as pure forward/reverse-mode approaches (since some subprograms that are intractable in one mode can be differentiated efficiently in the other).

I'm currently in the early stages of designing a markup language for specifying "differentiable signatures", defining mixed-mode chain rules, annotating functional properties like linearity etc. that will essentially be Capstan's version of DiffRules.jl. IMO, the primitive registration format laid out in the manifesto is quite interesting and highly related to this endeavor.