MachineLearningLifeScience / stochman

Algorithms for computations on random manifolds made easier
Apache License 2.0
85 stars 11 forks source link

Extend nnj functionalities #19

Closed FrederikWarburg closed 2 years ago

FrederikWarburg commented 2 years ago

In this pull request, we have made the following changes:

  1. _jacobian_mult is renamed to _jacobian_wrt_input_mult_left_vec
  2. We implement right multiplication for linear, conv2d, maxpool, upsampling, tanh layers. We refer to this as _jacobian_wrt_input_mult_right_vec
  3. We implement jacobian wrt. weights for linear and conv2d layers. We refer to these functions as _jacobian_wrt_weight_mult_right_vec and _jacobian_wrt_weight_mult_right_vec
  4. We implement the Generalized Gauss Newton (GGN) sandwich product wrt. weights and input for linear, conv2d, maxpool, upsampling, tanh layers. We refer to this as _jacobian_wrt_input_sandwich and _jacobian_wrt_weight_sandwich. We implement these GGN product for different approximations, namely diag_inp and diag_out referring to the approximations wrt. input and output respectively.