TODO

[GiggleLiu]

1. [x] discuss the relationship between NiLang and other reversible languages
2. [x] cite and discuss the relation between NiLang and other frameworks utilizing reversibility in ML. Checkpointing. (and explain what is checkpointing.)
3. [x] the flux, pytorch and tf benchmarks, explain why tf svd are manual.
4. [x] comment on the advantage towards tapenade
5. [ ] experiment support the memory efficiency argument
6. [ ] polish citation
7. [x] polish grammar
8. [x] move code to appendix
9. [x] explain why the reversible coding style matters
10. [x] remove introduction to AD
11. [x] compare NiLang with previous reversible languages
12. [x] show the example earlier.
13. [x] analyse the limits of the language. address: "It seems to me that the set of programs that this DSL can reverse is significantly smaller than the set of reversible programs that exist. What exactly is the relation between these two classes? Can you show this formally, or through examples of simple programs with a known inverse that you cannot express, and why?"
14. [x] explore the subtle commonalities and differences between reverse mode AD and reversible programming
15. [x] explain how to reproduce the benchmark results
16. [x] define "limited instruction autodiff" and "infite instruction autodiff"

[GiggleLiu]

• [ ] As the paper states at the end of page 2, it is not possible to rigorously reverse operations on floating point numbers. However, there is no further discussion of the implications of this. It would be good to have some further reassurance that the errors are not important, or to have a discussion of what applications are acceptable and sufficiently tolerant of the errors.

• [x] Given the lack of checkpointing required for automatic differentiation, it would seem to me that the language can enable significantly lower memory usage than non-reversible languages. So I was surprised that there are no benchmarks discussing memory usage, or showing that the method can operate with larger datasets/parameters on a fixed memory budget.

• [ ] ICLR is about machine learning, but there is no evaluation of machine learning workloads. For example, I think the ICLR audience would be quite interested in how such a system might enable very deep neural networks.

[GiggleLiu]