Parallel training of DNN

[nnop]

GPUDirect vs CUDA-Aware MPI

• http://timdettmers.com/2014/09/21/how-to-build-and-use-a-multi-gpu-system-for-deep-learning/
• http://timdettmers.com/2014/11/09/model-parallelism-deep-learning/
• https://devblogs.nvidia.com/parallelforall/introduction-cuda-aware-mpi/

[nnop]

Parallel training methods

• Alex Krizhevsky (2014). One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404.5997.
  • a paper reading note
• http://engineering.skymind.io/distributed-deep-learning-part-1-an-introduction-to-distributed-training-of-neural-networks
• https://www.nervanasys.com/faster-training-in-neon-with-multiple-gpus-on-the-nervana-cloud/

[nnop]

caffe data parallelism, multi-GPU

• Multi-GPU operation and data / model Parallelism #876
• Parallel / distributed training #1140
• Parallel / distributed training #1148
• Multi-GPU #2114
• Multi-GPU #2870
• Multi-GPU Data Parallelism (with Parallel Data Layers) #2903
• Merge NVIDIA's NCCL multi-GPU, switch it to python #4563
• Fix MultiGPU hangs when interrupting training #5825

[nnop]

• 如何理解Nvidia英伟达的Multi-GPU多卡通信框架NCCL？