search

noahgolmant / pytorch-hessian-eigenthings

Efficient PyTorch Hessian eigendecomposition tools!
MIT License
360 stars 43 forks source link

Create hessian tests on small networks, measure convergence vs hyperparameters #18

Closed noahgolmant closed 4 years ago

noahgolmant commented 5 years ago

(a) Test hessian computation on small networks where we can use np.linalg.eig to test against. Single hidden layer should be fine. (b) Measure convergence rate vs. batch size, power iteration steps. Inject noise to see how variance affects this. Change width of hidden layer, too. (c) Test averaging the eigenvalue estimate to see if this helps, too.

Should address #17

noahgolmant commented 5 years ago

(a) and (c) solved by #26

noahgolmant commented 4 years ago

From my experiences (b) seems to be heavily dependent on architecture, but overall I've been getting stable estimates on ResNet18 with as few as 10 steps, so I'm going to call it good for now