search

ctongfei / nexus

Experimental tensor-typed deep learning
https://tongfei.me/nexus/
MIT License
257 stars 15 forks source link

Underlying autograd mechanism #16

Open ctongfei opened 6 years ago

ctongfei commented 6 years ago

C. Elliott (2018): The simple essence of automatic differentiation. ICFP. ?

doofin commented 5 years ago

Do you use the reverse mode diff techniques in this paper? BTW,forward AD is easy to understand with dual numbers ,would you like to recommend some papers on reverse mode ad?

ctongfei commented 5 years ago

No. Right now nexus-diff uses Wengert tape-based mechanism (the same as PyTorch). The paper above is a candidate for a future implementation.

http://www.cs.cmu.edu/~wcohen/10-605/notes/autodiff.pdf -- this note on reverse-mode autodiff is pretty good