search

noahgolmant / pytorch-hessian-eigenthings

Efficient PyTorch Hessian eigendecomposition tools!
MIT License
360 stars 43 forks source link

Add continuous Hessian spectrum throughout training #7

Open noahgolmant opened 5 years ago

noahgolmant commented 5 years ago

Add GROUSE implementation for efficient low-rank subspace tracking with Grassmannian SGD based on the following paper (or other more recent subspace tracking papers): https://people.eecs.berkeley.edu/~brecht/papers/10.Bal.Now.Rec.GROUSE.pdf

noahgolmant commented 5 years ago

As a first step, just gonna try incrementally updating the eigenvector estimates using the previous computations as the initialization

themightyoarfish commented 5 years ago

Regarding GROUSE, there seems to be a Matlab implementation by the author, not sure if that's going to be helpful. I'd be curious if this can be used in a similar way to the SVCCA paper by google, but I probably won't get to that.

noahgolmant commented 5 years ago

It's been a while since I thought about this feature, but that link is certainly helpful. Looking back on it, the GROUSE implementation could be modified to solve the least squares objective through the conjugate gradient method. I think this should end up having the same computational complexity as power iteration.