Can I use torch.no_grad()?

choasma / HSIC-bottleneck

The HSIC Bottleneck: Deep Learning without Back-Propagation

https://arxiv.org/abs/1908.01580

MIT License

81 stars 17 forks source link

Can I use torch.no_grad()? #3

Closed zwep closed 4 years ago

zwep commented 4 years ago

Hi,

I was wondering something when I was reading you code. Since we are not using backprop in the usual sense, but updating layers on a layer-by-layer basis... can we turn off the standard gradient calculation that is done within torch?

This should, of course, save a lot of time.. and it seems to me that that should be the case. But I am not certain at this point.

choasma commented 4 years ago

Yeah, I think so. The current implementation is really pushing out the work out of the box and I've made dirty on it. It basically forces the other layer don't update by setting requires_grad False. Performance is awful in my implementation.

choasma commented 4 years ago

hi @zwep, please feel free to reopen this ticket or create new for any further question, thanks!