devmotion
commented
8 months ago I agree mostly, I think. Basically, as eg. mentioned in some other issues, my take is that I think AbstractDifferentiation should provide 1) a consistent interface for AD systems and 2) reasonable default implementations for eg jacobian based on pullback etc - but I think each backend (or currently extension) should define the interface explicitly without magic defaults. What I mean is that you should explicitly define whether to use an AbstractDifferentation default for a function or another existing functionality. I assume that backends might want to switch some of their existing functionality to such defaults if they treat AbstractDifferentiation as a first class dependency at some point; and the defaults are useful in early development stage if a package does not want to re-implement jacobian etc functionality. But currently we should try to be as consistent as possible with the AD packages and use their APIs whenever possible.
I increasingly feel like all the heavy lifting should be done by backends, and AbstractDifferentiation.jl should just define common names.
Whenever we do something that a backend does not support, or when we recode it instead of letting the package do it, we introduce correctness, readability or performance issues. Some examples:
Moreover, the complexity of the package also makes it very hard to push Enzyme support over the finish line (#84), which is arguably the biggest missing feature.
@adrhill and I are working on a prototype to play with this idea:
https://github.com/gdalle/DifferentiationInterface.jl
It may or may not be merged into AbstractDifferentiation, depending on what the team here thinks. The main caveat I see is that higher order cannot be well supported with an empty shell, but for gradients and JVPs/VJPs it should be close to optimal.