Agustin-Picard
commented
2 years ago I think it might be interesting if we could capture the amount of available memory (be it GPU or CPU, depending on the system's configuration) and optimize the computation method to better accomodate the resources.
Are we talking just memory/speed tradeoff or computation precision with the different methods as well? I'd think that if we have access to pod TPUs, we could maybe try to see the extent to which we can accurately calculate influence vectors/values on up to cifar10 networks.
In any case, I wouldn't think that integrating these benchmarks to the unit test suite would be interesting. Unit tests are mainly there to make sure that we didn't break anything whilst changing parts of the codebase, and our other tests should check this already.
So, yeah, I'm down for discussing this anytime @fel-thomas @lucashervier ;)
as discussed by @lucashervier, it would be good to have a way to properly benchmark our inverse hessian vector product calculator on different parameter dimensions -- mnist, cifar, imagenet convolution networks for example -- in order to have benchmarks on the amount of memory needed to build the inverse hessian in each case.
We still need to figure out if we integrate that in the test suite, or if we provide a notebook to benchmark ? We could meet at some point to discuss about it (@lucashervier, @Agustin-Picard). :)