Optimizer Design and related Operator

[jacquesqiao]

1. Implement various optimizer operators.
   1. SGD (done) and others listed in https://github.com/PaddlePaddle/Paddle/projects/22
2. Add optimizer operators into BlockDesc.
3. Python module

In the new framework with operators, we will use operators to build optimizer, and operators will be add to some block. There are some works to do:

• write a python wrapper of optimizer and provide proper interface for user to use.
• optimizer of multi GPU and multi machine is different then single device.
  • because it will be a seperate stage to update the parameters.
  • it will communicate with parameter server

there are three situation to be considered:

1. single machine single device.
2. single machine multiple devices.
3. multiple machine.

Plan:

• 8/24/2017 do some survey and discussion about tensorflow, caffe2 and pytorch on the python side. [done] https://github.com/PaddlePaddle/Paddle/issues/3672
• 8/25/2017 do some survey and discussion about distribute optimizers.
• 8/25/2017 design doc

distribute related issue: https://github.com/PaddlePaddle/Paddle/issues/3656

[jacquesqiao]

For tensorflow: https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py https://www.tensorflow.org/tutorials/deep_cnn

[jacquesqiao]

close this old one and open a new issue: https://github.com/PaddlePaddle/Paddle/issues/4679