Optimize pytorch run as good as possible.

[vegovs]

• [x] Asynchronous data loading and augmentation enabled.
• [x] cuDNN autotuner enabled.
• [x] Automatic mixed precision, i.e use of float16 in tensorcores.
• [ ] Increase batch-size due to reduced memory usage.
  • [ ] Tune learning rate.
  • [ ] Add learning rate warmup.
  • [x] Add learning decay.
  • [ ] Add weight decay.
  • [ ] Use optimizer for large-barch training(LARS, LAMB, NVLAMB, NovoGrad).
• [x] Disable bias for convolutions directly followed by a batch norm.
• [x] Use parameter.grad = None.
• [x] Disbale debut APIs in production-mode.
• [ ] Opt for DistributedDataParallel instead of DataParallel to improve Multi-GPU efficiency.
  • [ ] Load-balance workload on multiple GPUs.
• [ ] Use fused building blocs from APEX.
• [ ] Checkpoint to recompute intermediates.
• [ ] Fuse pointwise operations.