search

tensorflow / kfac

An implementation of KFAC for TensorFlow
Apache License 2.0
197 stars 41 forks source link

Where are natural gradients for convolutional layers renormalized by num_locations #36

Closed xmax1 closed 4 years ago

xmax1 commented 4 years ago

Where in the code are the natural gradients for the convolutional layers renormalized by _renorm_coeff (num_locations)?

Is this handled by the tensorflow optimizer class after the inverse covariances are applied to the gradients? Otherwise, I can't find this renormalization in the compute_cov / compute_inv / apply gradients functions.

It might be a bug, though this doesn't seem like the right conclusion given the effect this would have on performance.

Thanks,

Max

xmax1 commented 4 years ago

Update:

Ok found it Line 864 fisher_blocks

Scratch this