Small optimisations - Githubissues

feldberlin / wavenet

An unconditioned Wavenet implementation with fast generation.

3 stars 0 forks source link

Small optimisations #21

Closed purzelrakete closed 3 years ago

purzelrakete commented 3 years ago

What

Follow the pytorch tuning guidelines from here, as recommended by NVIDIA. Plus some others.

Why

Will yield a small improvement in training times. Super painless, so why not

Acceptance Criteria

[x] Enable cudnn autotuner
[x] Use efficient gradient zeroing
[x] Look for pointwise ops to fuse with pytorch JIT

purzelrakete commented 3 years ago

Results

These small tweaks have sped training up by a factor of 2x, down to 6 hours for 12 epochs, where before it was 12 hours for 12 epochs. This is a much larger improvement than what I expected.