search

tfjgeorge / nngeometry

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch
https://nngeometry.readthedocs.io
MIT License
206 stars 20 forks source link

Montecarlo Sampling Question #32

Closed simomagi closed 3 years ago

simomagi commented 3 years ago

Looking at the implementation of the Monte Carlo Sampling for computing the Fisher Information I see that the sampled probabilities are divided by the square root of the number of trials. Is there a specific reason for this? Why not simply dividing by the number of trials?

tfjgeorge commented 3 years ago

Hi, if you look at the formula: Screenshot from 2021-10-01 15-05-50

In order to get a 1/n on the left hand side, you need to factorize by 1/sqrt(n) in the parenthesis on the right hand side