why the train loop is very very long time ?

devzhk / Implicit-Competitive-Regularization

Code for: Implicit Competitive Regularization in GANs

Apache License 2.0

114 stars 18 forks source link

why the train loop is very very long time ? #7

Closed Johnson-yue closed 4 years ago

Johnson-yue commented 4 years ago

I think ACGD is cost very long time in train ,why ?

devzhk commented 4 years ago

Yes, it usually takes 4x-6x longer than Adam. Because ACGD has Conjugate Gradient and Hessian vector product inside.

devzhk commented 4 years ago

Part of reason can be that the code is not optimal. Efficiency of the algorithm is one important future work. What's more, some parts are limited by Pytorch framework. For example, Pytorch only supports backprop but the most efficient way to do Hessian vector product is to combine backprop and forward mode autodiff.

devzhk commented 4 years ago

As a reminder, it's recommended to set torch.backends.cudnn.benchmark = True. Sometimes cudnn heuristics picks truly atrocious algorithm for couple layers which may make ACGD training super slow.

zr-bee commented 1 year ago

Can it be faster?