Add neural network gradient tests [Juliacon 2024 Hackathon]

[gdalle]

Goal

High-level: Support structures beyond arrays and numbers in DifferentiationInterface.

Low-level: Write tests for taking gradients of neural networks with Flux and Lux.

Steps

1. Read the slides of my autodiff intro https://gdalle.github.io/JuliaCon2024-AutoDiff/#/title-slide
2. Read some of the DifferentiationInterface documentation to understand the general ideas:
   • DI tutorial
   • DI operators
   • DITest tutorial
   • DITest Scenario API
3. Read DITest source code to understand how test scenarios are defined
4. Fork the DifferentiationInterface repository and open a pull request
5. Add a file to DifferentiationInterfaceTest/src/scenarios called flux.jl
6. Define a GradientScenario involving a very simple neural network built with Flux.jl, for instance the one in this tutorial.
7. Usually in deep learning we differentiate with respect to the parameters of a layer. In Flux, these parameters are stored within the layer itself. So the gradient we need is the gradient of layer(input) with respect to layer!!! In other words, for your GradientScenario, you will have f(layer) = layer(fixed_input) as the function (it only applies the layer to a fixed input).
8. Add a file to DifferentiationInterface/test/Single/Zygote called flux.jl and test your scenario with DifferentiationInterfaceTest.test_differentiation. Take inspiration from the other test files.
9. Since layer is not an array, the returned type will not be an array either: the gradient will be some form of Flux layer as well (I think), so you probably want to compute the ground truth with Zygote at first to see how it is structured.

If you need help

• Come find me (@gdalle)
• Check out https://modernjuliaworkflows.github.io/ for the Julia development tricks
• Check out https://kshyatt.github.io/post/firstjuliapr/ for the GitHub-related issues

Participants

• @nialamarcotte

[nialamarcotte]

Thanks for the experience.

I'm still traveling but I hope to look into what you did afterwards, no promises though. See you likely at the next JuliaCon (global).

Alain Marcotte

Avant tout le respect: de soi, des autres, de l'environnement

Le mer. 17 juil. 2024 à 10:14, Guillaume Dalle @.***> a écrit :

Closed #346 https://github.com/gdalle/DifferentiationInterface.jl/issues/346 as completed via #352 https://github.com/gdalle/DifferentiationInterface.jl/pull/352.

— Reply to this email directly, view it on GitHub https://github.com/gdalle/DifferentiationInterface.jl/issues/346#event-13532748134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOAF5ZOSD7AY7QH7EGFQYLZMYRXFAVCNFSM6AAAAABK2CZXAOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGUZTENZUHAYTGNA . You are receiving this because you were mentioned.Message ID: <gdalle/DifferentiationInterface.jl/issue/346/issue_event/13532748134@ github.com>