Minor performance tweaks to match the guide.

[tfboyd]

I believe these are still NHWC which I did not want to mess with. I fixed the following:

• Input pipeline on CPU
• batch_norm fused
• for AlexNet I moved the local_ps_device to GPU which is the better option if the GPUs are peered and about equal to CPU:0 otherwise. No fancy NCCL which does not seem to work well on K80s and no non-NCCL replicated variables which would likely speed this up as well.
• I added a flag to both for local_ps_device and set it to the default that I expect works the best on K80s and likely P100s as well.
• I fixed the eval to set is_training = False. Does not matter for perf testing but that is how it is supposed to be.
• ResNet also converges to a slightly better loss now due to a change in the learning rate. It could be better but I was not interested in adding a learning rate decay right now.
• AlexNet converges to the same level as before but I think a little faster but that might be luck.
• I noticed your K80 is clocked at ~500Mhz. The normal is ~845Mhz that seemed weird but I assume consistent.
• I also did some general cleanup where the losses were added to a collection and then taken out and then added to nothing and then I am not sure what it was but it was not needed.
  Finally, off the top of my head the numbers I see:
• AlexNet 4 GPUs ~70ms on AWS with 4 K80s using the CPU as the PS_server this is ~100ms
• ResNet 4 GPUs ~140ms on AWS and Google Cloud
• Both are I think 50% or more savings in time or 2x speedup or whatever.

Some people think that the batch_norm is not doing beta and gamma. Check the code it is built into the function you should do not control how it is initialized.

Finally, the changes I made are not esoteric. I just added the basic that I would suggest for anyone else. I might change it over to NCHW but this seemed good enough for now. I think we are about to push a "crazy" alexnet in our benchmark that uses some fun staging ops but I did not want to mess up your scripts.

[tfboyd]

thank you @shyhuai If you have any questions just ask. This is not perfect but a little better without too much work.

[FreemanX]

Thank you for contributing to the project!

[shyhuai]

@tfboyd many thanks for your contribution and the detailed comments in improving the codes. We will update the results to our website soon.