mxnet benchmark

[pineking]

结论

• 单机多卡与单机单卡相比，基本线性增加
• 多机（2机）多卡与单机单卡相比，基本线性增加
• 单机八卡中，前四卡可以 P2P，后四卡可以 P2P，前四卡和后四卡没有 P2P
• cifar10 实验：默认 batch size =128，GPU 占用率 90%，提高 batch size =256 , GPU 占用率提高到 95 %，但 batch size 太大会减慢收敛速度，对于 cifar10 ，batch size = 200 合适

	batch size	秒/Epoch	GPU 占用率
单机单卡	128	51	90%
单机单卡	256	44	95%
单机两卡	256	24	90%
单机四卡	512	12	90%
单机八卡	1024	7.0	80%
两机，每机四卡	512	6.5	87%

机器配置

• 2台机器，机器配置相同，GPU 均为 GTX TITAN X ，每台机器 8 卡
• CentOS 7.2.1511 + nvidia driver 375.20 + CUDA 8.0 + cudnn 5.1 + openblas
• mxnet: 20161211 commit 962271410059156180ab1d5e79b805e687512be9

数据与网络

• 采用 cifar10 数据库，5w examples，分10类
• 网络结构采用 resnet 101层，参加文件

单机单卡

• 默认 batch-size=128 , GPU 占用率 90% 左右，显存占用 1455MB

python train_cifar10.py --gpus=0
INFO:root:start with arguments Namespace(batch_size=128, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
[18:20:17] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
[18:20:17] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
INFO:root:Start training with [gpu(0)]
INFO:root:Epoch[0] Batch [20]   Speed: 939.26 samples/sec       Train-accuracy=0.128125
INFO:root:Epoch[0] Batch [40]   Speed: 984.92 samples/sec       Train-accuracy=0.205859
INFO:root:Epoch[0] Batch [60]   Speed: 983.00 samples/sec       Train-accuracy=0.219922
INFO:root:Epoch[0] Batch [80]   Speed: 978.03 samples/sec       Train-accuracy=0.229687
INFO:root:Epoch[0] Batch [100]  Speed: 982.58 samples/sec       Train-accuracy=0.253125
INFO:root:Epoch[0] Batch [120]  Speed: 963.30 samples/sec       Train-accuracy=0.263281
INFO:root:Epoch[0] Batch [140]  Speed: 998.05 samples/sec       Train-accuracy=0.297266
INFO:root:Epoch[0] Batch [160]  Speed: 978.65 samples/sec       Train-accuracy=0.308203
INFO:root:Epoch[0] Batch [180]  Speed: 984.20 samples/sec       Train-accuracy=0.331250
INFO:root:Epoch[0] Batch [200]  Speed: 995.25 samples/sec       Train-accuracy=0.327344
INFO:root:Epoch[0] Batch [220]  Speed: 988.98 samples/sec       Train-accuracy=0.332813
INFO:root:Epoch[0] Batch [240]  Speed: 993.27 samples/sec       Train-accuracy=0.353906
INFO:root:Epoch[0] Batch [260]  Speed: 991.30 samples/sec       Train-accuracy=0.363672
INFO:root:Epoch[0] Batch [280]  Speed: 987.99 samples/sec       Train-accuracy=0.367969
INFO:root:Epoch[0] Batch [300]  Speed: 990.45 samples/sec       Train-accuracy=0.380469
INFO:root:Epoch[0] Batch [320]  Speed: 984.57 samples/sec       Train-accuracy=0.397656
INFO:root:Epoch[0] Batch [340]  Speed: 999.65 samples/sec       Train-accuracy=0.429297
INFO:root:Epoch[0] Batch [360]  Speed: 995.40 samples/sec       Train-accuracy=0.425000
INFO:root:Epoch[0] Batch [380]  Speed: 987.50 samples/sec       Train-accuracy=0.421094
INFO:root:Epoch[0] Resetting Data Iterator
INFO:root:Epoch[0] Time cost=51.785
INFO:root:Epoch[0] Validation-accuracy=0.443236
INFO:root:Epoch[1] Batch [20]   Speed: 1041.97 samples/sec      Train-accuracy=0.443359
INFO:root:Epoch[1] Batch [40]   Speed: 991.77 samples/sec       Train-accuracy=0.483203
INFO:root:Epoch[1] Batch [60]   Speed: 1008.41 samples/sec      Train-accuracy=0.446094
INFO:root:Epoch[1] Batch [80]   Speed: 1004.69 samples/sec      Train-accuracy=0.471484
INFO:root:Epoch[1] Batch [100]  Speed: 1001.01 samples/sec      Train-accuracy=0.484375
INFO:root:Epoch[1] Batch [120]  Speed: 998.18 samples/sec       Train-accuracy=0.501953
INFO:root:Epoch[1] Batch [140]  Speed: 980.64 samples/sec       Train-accuracy=0.498828
INFO:root:Epoch[1] Batch [160]  Speed: 1003.91 samples/sec      Train-accuracy=0.535547
INFO:root:Epoch[1] Batch [180]  Speed: 1000.82 samples/sec      Train-accuracy=0.544531
INFO:root:Epoch[1] Batch [200]  Speed: 992.89 samples/sec       Train-accuracy=0.530078
INFO:root:Epoch[1] Batch [220]  Speed: 1010.66 samples/sec      Train-accuracy=0.514453
INFO:root:Epoch[1] Batch [240]  Speed: 1005.17 samples/sec      Train-accuracy=0.535547
INFO:root:Epoch[1] Batch [260]  Speed: 988.56 samples/sec       Train-accuracy=0.534766
INFO:root:Epoch[1] Batch [280]  Speed: 989.83 samples/sec       Train-accuracy=0.545703
INFO:root:Epoch[1] Batch [300]  Speed: 1003.98 samples/sec      Train-accuracy=0.557031

• batch-size=256 , GPU 占用率 95% 左右，显存占用 2600+MB

python train_cifar10.py --gpus=0 --batch-size=256
INFO:root:start with arguments Namespace(batch_size=256, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
[18:24:29] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
[18:24:29] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
INFO:root:Start training with [gpu(0)]
INFO:root:Epoch[0] Batch [20]   Speed: 1232.52 samples/sec      Train-accuracy=0.153320
INFO:root:Epoch[0] Batch [40]   Speed: 1151.11 samples/sec      Train-accuracy=0.232422
INFO:root:Epoch[0] Batch [60]   Speed: 1158.12 samples/sec      Train-accuracy=0.259766
INFO:root:Epoch[0] Batch [80]   Speed: 1141.59 samples/sec      Train-accuracy=0.278906
INFO:root:Epoch[0] Batch [100]  Speed: 1147.25 samples/sec      Train-accuracy=0.301758
INFO:root:Epoch[0] Batch [120]  Speed: 1150.68 samples/sec      Train-accuracy=0.317969
INFO:root:Epoch[0] Batch [140]  Speed: 1148.73 samples/sec      Train-accuracy=0.349414
INFO:root:Epoch[0] Batch [160]  Speed: 1144.52 samples/sec      Train-accuracy=0.352539
INFO:root:Epoch[0] Batch [180]  Speed: 1141.29 samples/sec      Train-accuracy=0.379102
INFO:root:Epoch[0] Resetting Data Iterator
INFO:root:Epoch[0] Time cost=44.563
INFO:root:Epoch[0] Validation-accuracy=0.433105
INFO:root:Epoch[1] Batch [20]   Speed: 1206.70 samples/sec      Train-accuracy=0.398438
INFO:root:Epoch[1] Batch [40]   Speed: 1145.39 samples/sec      Train-accuracy=0.412109
INFO:root:Epoch[1] Batch [60]   Speed: 1144.15 samples/sec      Train-accuracy=0.430273
INFO:root:Epoch[1] Batch [80]   Speed: 1142.66 samples/sec      Train-accuracy=0.441211
INFO:root:Epoch[1] Batch [100]  Speed: 1141.73 samples/sec      Train-accuracy=0.458203
INFO:root:Epoch[1] Batch [120]  Speed: 1143.65 samples/sec      Train-accuracy=0.470508
INFO:root:Epoch[1] Batch [140]  Speed: 1142.52 samples/sec      Train-accuracy=0.489258
INFO:root:Epoch[1] Batch [160]  Speed: 1140.67 samples/sec      Train-accuracy=0.513672
INFO:root:Epoch[1] Batch [180]  Speed: 1138.96 samples/sec      Train-accuracy=0.531445
INFO:root:Epoch[1] Resetting Data Iterator
INFO:root:Epoch[1] Time cost=43.542
INFO:root:Epoch[1] Validation-accuracy=0.547070
INFO:root:Epoch[2] Batch [20]   Speed: 1212.84 samples/sec      Train-accuracy=0.547461
INFO:root:Epoch[2] Batch [40]   Speed: 1140.71 samples/sec      Train-accuracy=0.553906

单机多卡

单机两卡

• batch-size=256 , GPU 占用率 90% 左右，显存占用 1466MB 左右

  python train_cifar10.py --gpus=0,1 --batch-size=256
  INFO:root:start with arguments Namespace(batch_size=256, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0,1', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
  [18:29:12] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
  [18:29:12] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
  INFO:root:Start training with [gpu(0), gpu(1)]
  INFO:root:Epoch[0] Batch [20]   Speed: 1930.93 samples/sec      Train-accuracy=0.160547
  INFO:root:Epoch[0] Batch [40]   Speed: 2011.66 samples/sec      Train-accuracy=0.216211
  INFO:root:Epoch[0] Batch [60]   Speed: 2008.91 samples/sec      Train-accuracy=0.258789
  INFO:root:Epoch[0] Batch [80]   Speed: 2032.00 samples/sec      Train-accuracy=0.277734
  INFO:root:Epoch[0] Batch [100]  Speed: 2004.61 samples/sec      Train-accuracy=0.304297
  INFO:root:Epoch[0] Batch [120]  Speed: 2034.65 samples/sec      Train-accuracy=0.305859
  INFO:root:Epoch[0] Batch [140]  Speed: 1978.84 samples/sec      Train-accuracy=0.329102
  INFO:root:Epoch[0] Batch [160]  Speed: 2007.93 samples/sec      Train-accuracy=0.338281
  INFO:root:Epoch[0] Batch [180]  Speed: 2008.85 samples/sec      Train-accuracy=0.366211
  INFO:root:Epoch[0] Resetting Data Iterator
  INFO:root:Epoch[0] Time cost=26.961
  INFO:root:Epoch[0] Validation-accuracy=0.424609
  INFO:root:Epoch[1] Batch [20]   Speed: 2149.68 samples/sec      Train-accuracy=0.390234
  INFO:root:Epoch[1] Batch [40]   Speed: 2031.56 samples/sec      Train-accuracy=0.395508
  INFO:root:Epoch[1] Batch [60]   Speed: 2021.19 samples/sec      Train-accuracy=0.416016
  INFO:root:Epoch[1] Batch [80]   Speed: 2023.69 samples/sec      Train-accuracy=0.441016
  INFO:root:Epoch[1] Batch [100]  Speed: 2007.46 samples/sec      Train-accuracy=0.454883
  INFO:root:Epoch[1] Batch [120]  Speed: 1962.99 samples/sec      Train-accuracy=0.455859
  INFO:root:Epoch[1] Batch [140]  Speed: 1997.71 samples/sec      Train-accuracy=0.475000
  INFO:root:Epoch[1] Batch [160]  Speed: 2006.17 samples/sec      Train-accuracy=0.485547
  INFO:root:Epoch[1] Batch [180]  Speed: 2022.69 samples/sec      Train-accuracy=0.512891
  INFO:root:Epoch[1] Resetting Data Iterator
  INFO:root:Epoch[1] Time cost=24.786
  INFO:root:Epoch[1] Validation-accuracy=0.522461
  INFO:root:Epoch[2] Batch [20]   Speed: 2167.01 samples/sec      Train-accuracy=0.531641
  INFO:root:Epoch[2] Batch [40]   Speed: 1993.56 samples/sec      Train-accuracy=0.537109
  INFO:root:Epoch[2] Batch [60]   Speed: 2001.09 samples/sec      Train-accuracy=0.554492
  INFO:root:Epoch[2] Batch [80]   Speed: 2022.72 samples/sec      Train-accuracy=0.546094
  INFO:root:Epoch[2] Batch [100]  Speed: 1982.08 samples/sec      Train-accuracy=0.562695
  INFO:root:Epoch[2] Batch [120]  Speed: 1985.08 samples/sec      Train-accuracy=0.575391
  INFO:root:Epoch[2] Batch [140]  Speed: 1996.59 samples/sec      Train-accuracy=0.571289
  INFO:root:Epoch[2] Batch [160]  Speed: 2031.67 samples/sec      Train-accuracy=0.586523
  INFO:root:Epoch[2] Batch [180]  Speed: 1996.78 samples/sec      Train-accuracy=0.602930
  INFO:root:Epoch[2] Resetting Data Iterator
  INFO:root:Epoch[2] Time cost=24.871
  INFO:root:Epoch[2] Validation-accuracy=0.624512

  单机四卡

• batch-size=512 , GPU 占用率 90% 左右，显存占用 1466MB 左右

  python train_cifar10.py --gpus=0,1,2,3 --batch-size=512
  INFO:root:start with arguments Namespace(batch_size=512, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0,1,2,3', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
  [18:33:52] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
  [18:33:52] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
  INFO:root:Start training with [gpu(0), gpu(1), gpu(2), gpu(3)]
  INFO:root:Epoch[0] Batch [20]   Speed: 3872.94 samples/sec      Train-accuracy=0.168066
  INFO:root:Epoch[0] Batch [40]   Speed: 4004.03 samples/sec      Train-accuracy=0.241602
  INFO:root:Epoch[0] Batch [60]   Speed: 4026.89 samples/sec      Train-accuracy=0.275586
  INFO:root:Epoch[0] Batch [80]   Speed: 3936.01 samples/sec      Train-accuracy=0.301172
  INFO:root:Epoch[0] Resetting Data Iterator
  INFO:root:Epoch[0] Time cost=16.976
  INFO:root:Epoch[0] Validation-accuracy=0.370215
  INFO:root:Epoch[1] Batch [20]   Speed: 4360.50 samples/sec      Train-accuracy=0.350977
  INFO:root:Epoch[1] Batch [40]   Speed: 4029.03 samples/sec      Train-accuracy=0.386035
  INFO:root:Epoch[1] Batch [60]   Speed: 4089.17 samples/sec      Train-accuracy=0.403516
  INFO:root:Epoch[1] Batch [80]   Speed: 3941.35 samples/sec      Train-accuracy=0.430371
  INFO:root:Epoch[1] Resetting Data Iterator
  INFO:root:Epoch[1] Time cost=12.365
  INFO:root:Epoch[1] Validation-accuracy=0.512988
  INFO:root:Epoch[2] Batch [20]   Speed: 4373.76 samples/sec      Train-accuracy=0.490234
  INFO:root:Epoch[2] Batch [40]   Speed: 4017.80 samples/sec      Train-accuracy=0.504785
  INFO:root:Epoch[2] Batch [60]   Speed: 3999.35 samples/sec      Train-accuracy=0.525684
  INFO:root:Epoch[2] Batch [80]   Speed: 3975.62 samples/sec      Train-accuracy=0.544043
  INFO:root:Epoch[2] Resetting Data Iterator
  INFO:root:Epoch[2] Time cost=12.279
  INFO:root:Epoch[2] Validation-accuracy=0.577148
  INFO:root:Epoch[3] Batch [20]   Speed: 4332.04 samples/sec      Train-accuracy=0.576758
  INFO:root:Epoch[3] Batch [40]   Speed: 4080.77 samples/sec      Train-accuracy=0.576855
  INFO:root:Epoch[3] Batch [60]   Speed: 3943.68 samples/sec      Train-accuracy=0.589648
  INFO:root:Epoch[3] Batch [80]   Speed: 3833.84 samples/sec      Train-accuracy=0.607812
  INFO:root:Epoch[3] Resetting Data Iterator
  INFO:root:Epoch[3] Time cost=12.714
  INFO:root:Epoch[3] Validation-accuracy=0.624609
  INFO:root:Epoch[4] Batch [20]   Speed: 4129.53 samples/sec      Train-accuracy=0.613086
  INFO:root:Epoch[4] Batch [40]   Speed: 3839.61 samples/sec      Train-accuracy=0.626660
  INFO:root:Epoch[4] Batch [60]   Speed: 3912.95 samples/sec      Train-accuracy=0.633887
  INFO:root:Epoch[4] Batch [80]   Speed: 3909.16 samples/sec      Train-accuracy=0.645215
  INFO:root:Epoch[4] Resetting Data Iterator
  INFO:root:Epoch[4] Time cost=12.909
  INFO:root:Epoch[4] Validation-accuracy=0.675586

  单机八卡

• batch-size=1024 , GPU 占用率 80% 左右，显存占用 1466MB 左右

  python train_cifar10.py --gpus=0,1,2,3,4,5,6,7 --batch-size=1024
  INFO:root:start with arguments Namespace(batch_size=1024, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0,1,2,3,4,5,6,7', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
  [18:38:32] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
  [18:38:33] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
  INFO:root:Start training with [gpu(0), gpu(1), gpu(2), gpu(3), gpu(4), gpu(5), gpu(6), gpu(7)]
  [18:39:03] src/kvstore/././comm.h:290: only 24 out of 56 GPU pairs are enabled direct access. It may affect the performance. You can set MXNET_ENABLE_GPU_P2P=0 to turn it off
  [18:39:03] src/kvstore/././comm.h:299: .vvv....
  [18:39:03] src/kvstore/././comm.h:299: v.vv....
  [18:39:03] src/kvstore/././comm.h:299: vv.v....
  [18:39:03] src/kvstore/././comm.h:299: vvv.....
  [18:39:03] src/kvstore/././comm.h:299: .....vvv
  [18:39:03] src/kvstore/././comm.h:299: ....v.vv
  [18:39:03] src/kvstore/././comm.h:299: ....vv.v
  [18:39:03] src/kvstore/././comm.h:299: ....vvv.
  INFO:root:Epoch[0] Batch [20]   Speed: 6940.86 samples/sec      Train-accuracy=0.174463
  INFO:root:Epoch[0] Batch [40]   Speed: 7017.27 samples/sec      Train-accuracy=0.254932
  INFO:root:Epoch[0] Resetting Data Iterator
  INFO:root:Epoch[0] Time cost=15.082
  INFO:root:Epoch[0] Validation-accuracy=0.323047
  INFO:root:Epoch[1] Batch [20]   Speed: 7747.81 samples/sec      Train-accuracy=0.314697
  INFO:root:Epoch[1] Batch [40]   Speed: 7190.83 samples/sec      Train-accuracy=0.349121
  INFO:root:Epoch[1] Resetting Data Iterator
  INFO:root:Epoch[1] Time cost=6.909
  INFO:root:Epoch[1] Validation-accuracy=0.376660
  INFO:root:Epoch[2] Batch [20]   Speed: 7659.83 samples/sec      Train-accuracy=0.394727
  INFO:root:Epoch[2] Batch [40]   Speed: 7188.02 samples/sec      Train-accuracy=0.426758
  INFO:root:Epoch[2] Resetting Data Iterator
  INFO:root:Epoch[2] Time cost=7.007
  INFO:root:Epoch[2] Validation-accuracy=0.447852
  INFO:root:Epoch[3] Batch [20]   Speed: 7685.18 samples/sec      Train-accuracy=0.474316
  INFO:root:Epoch[3] Batch [40]   Speed: 7005.55 samples/sec      Train-accuracy=0.502197
  INFO:root:Epoch[3] Resetting Data Iterator
  INFO:root:Epoch[3] Time cost=7.026
  INFO:root:Epoch[3] Validation-accuracy=0.552344
  INFO:root:Epoch[4] Batch [20]   Speed: 7753.11 samples/sec      Train-accuracy=0.548486
  INFO:root:Epoch[4] Batch [40]   Speed: 7059.63 samples/sec      Train-accuracy=0.561328
  INFO:root:Epoch[4] Resetting Data Iterator
  INFO:root:Epoch[4] Time cost=6.988
  INFO:root:Epoch[4] Validation-accuracy=0.604102
  INFO:root:Epoch[5] Batch [20]   Speed: 7640.04 samples/sec      Train-accuracy=0.595654

多机多卡

两机，每机四卡

• batch-size=1024 , GPU 占用率 88% 左右，显存占用 1448MB 左右

  [root@0c-c4-7a-82-c5-bc image-classification]# ../../tools/launch.py -n 2 --launcher ssh -H  hosts python train_cifar10.py --batch-size=512 --gpus=0,1,2,3 --kv-store dist_sync
  Warning: Permanently added '0c-c4-7a-82-c5-b8,10.10.14.66' (ECDSA) to the list of known hosts.
  Warning: Permanently added '0c-c4-7a-82-c5-b8,10.10.14.66' (ECDSA) to the list of known hosts.
  Warning: Permanently added '0c-c4-7a-82-c5-bc,10.10.14.70' (ECDSA) to the list of known hosts.
  Warning: Permanently added '0c-c4-7a-82-c5-bc,10.10.14.70' (ECDSA) to the list of known hosts.
  INFO:root:start with arguments Namespace(batch_size=512, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0,1,2,3', image_shape='3,28,28', kv_store='dist_sync', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
  INFO:root:start with arguments Namespace(batch_size=512, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, gpus='0,1,2,3', image_shape='3,28,28', kv_store='dist_sync', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
  [18:50:02] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
  [18:50:02] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..
  [18:50:02] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
  [18:50:02] src/io/iter_image_recordio.cc:221: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..
  INFO:root:Start training with [gpu(0), gpu(1), gpu(2), gpu(3)]
  INFO:root:Start training with [gpu(0), gpu(1), gpu(2), gpu(3)]
  INFO:root:Epoch[0] Batch [20]   Speed: 4360.74 samples/sec      Train-accuracy=0.164648
  INFO:root:Epoch[0] Batch [20]   Speed: 4516.36 samples/sec      Train-accuracy=0.158594
  INFO:root:Epoch[0] Batch [40]   Speed: 4122.36 samples/sec      Train-accuracy=0.237207
  INFO:root:Epoch[0] Batch [40]   Speed: 4123.93 samples/sec      Train-accuracy=0.240234
  INFO:root:Epoch[0] Resetting Data Iterator
  INFO:root:Epoch[0] Resetting Data Iterator
  INFO:root:Epoch[0] Time cost=6.367
  INFO:root:Epoch[0] Time cost=6.377
  INFO:root:Epoch[0] Validation-accuracy=0.307617
  INFO:root:Epoch[0] Validation-accuracy=0.301367
  INFO:root:Epoch[1] Batch [20]   Speed: 4400.07 samples/sec      Train-accuracy=0.308594
  INFO:root:Epoch[1] Batch [20]   Speed: 4392.16 samples/sec      Train-accuracy=0.297559
  INFO:root:Epoch[1] Batch [40]   Speed: 4195.24 samples/sec      Train-accuracy=0.330273
  INFO:root:Epoch[1] Batch [40]   Speed: 4194.41 samples/sec      Train-accuracy=0.329199
  INFO:root:Epoch[1] Resetting Data Iterator
  INFO:root:Epoch[1] Time cost=5.907
  INFO:root:Epoch[1] Resetting Data Iterator
  INFO:root:Epoch[1] Time cost=5.911
  INFO:root:Epoch[1] Validation-accuracy=0.355273
  INFO:root:Epoch[1] Validation-accuracy=0.344727
  ...
  INFO:root:Epoch[18] Batch [40]  Speed: 3788.69 samples/sec      Train-accuracy=0.787793
  INFO:root:Epoch[18] Resetting Data Iterator
  INFO:root:Epoch[18] Time cost=6.546
  INFO:root:Epoch[18] Resetting Data Iterator
  INFO:root:Epoch[18] Time cost=6.567
  INFO:root:Epoch[18] Validation-accuracy=0.780859
  INFO:root:Epoch[18] Validation-accuracy=0.792969
  INFO:root:Epoch[19] Batch [20]  Speed: 3965.41 samples/sec      Train-accuracy=0.804980
  INFO:root:Epoch[19] Batch [20]  Speed: 3987.19 samples/sec      Train-accuracy=0.798535
  INFO:root:Epoch[19] Batch [40]  Speed: 3778.22 samples/sec      Train-accuracy=0.797266
  INFO:root:Epoch[19] Batch [40]  Speed: 3776.28 samples/sec      Train-accuracy=0.803516
  INFO:root:Epoch[19] Resetting Data Iterator
  INFO:root:Epoch[19] Time cost=6.541
  INFO:root:Epoch[19] Resetting Data Iterator
  INFO:root:Epoch[19] Time cost=6.555
  INFO:root:Epoch[19] Validation-accuracy=0.795898
  INFO:root:Epoch[19] Validation-accuracy=0.803516
  INFO:root:Epoch[20] Batch [20]  Speed: 3976.12 samples/sec      Train-accuracy=0.805566
  INFO:root:Epoch[20] Batch [20]  Speed: 3984.60 samples/sec      Train-accuracy=0.801270
  INFO:root:Epoch[20] Batch [40]  Speed: 3785.69 samples/sec      Train-accuracy=0.805176
  INFO:root:Epoch[20] Batch [40]  Speed: 3785.16 samples/sec      Train-accuracy=0.806641
  INFO:root:Epoch[20] Resetting Data Iterator
  INFO:root:Epoch[20] Time cost=6.535
  INFO:root:Epoch[20] Resetting Data Iterator
  INFO:root:Epoch[20] Time cost=6.540
  INFO:root:Epoch[20] Validation-accuracy=0.791797
  INFO:root:Epoch[20] Validation-accuracy=0.799219
  INFO:root:Epoch[21] Batch [20]  Speed: 3966.76 samples/sec      Train-accuracy=0.801953
  INFO:root:Epoch[21] Batch [20]  Speed: 3979.63 samples/sec      Train-accuracy=0.800977
  INFO:root:Epoch[21] Batch [40]  Speed: 3781.53 samples/sec      Train-accuracy=0.809082
  INFO:root:Epoch[21] Batch [40]  Speed: 3780.61 samples/sec      Train-accuracy=0.814355
  INFO:root:Epoch[21] Resetting Data Iterator
  INFO:root:Epoch[21] Time cost=6.557
  INFO:root:Epoch[21] Resetting Data Iterator
  INFO:root:Epoch[21] Time cost=6.550
  INFO:root:Epoch[21] Validation-accuracy=0.769336
  INFO:root:Epoch[21] Validation-accuracy=0.757227
  INFO:root:Epoch[22] Batch [20]  Speed: 3966.17 samples/sec      Train-accuracy=0.821094
  INFO:root:Epoch[22] Batch [20]  Speed: 3981.77 samples/sec      Train-accuracy=0.815820
  INFO:root:Epoch[22] Batch [40]  Speed: 3769.84 samples/sec      Train-accuracy=0.810742
  INFO:root:Epoch[22] Batch [40]  Speed: 3772.21 samples/sec      Train-accuracy=0.822363
  INFO:root:Epoch[22] Resetting Data Iterator
  INFO:root:Epoch[22] Time cost=6.542
  INFO:root:Epoch[22] Resetting Data Iterator
  INFO:root:Epoch[22] Time cost=6.556
  INFO:root:Epoch[22] Validation-accuracy=0.798242
  INFO:root:Epoch[22] Validation-accuracy=0.804688

[xuerq]

补充测试了mxnet 在 k8s 集群上的分布式性能

• 2台机器，每台机器8个gpu
• 分别测试了，单机多卡、分布式单机多卡、分布式多机多卡
• 分别测试了 batch_size=128 、batch_size=256 两种情况
• 从数据中可以看到，mxnet 在 k8s 上的分布式多机多卡性能很接近，单机多卡，batch_size=256时，尤为明显

batch_size=128

batch_size=256

[pineking]

@xuerq 关于分布式 mxnet 的多机多卡实验

• 同等配置下，在 k8s 中运行的分布式和 mxnet 直接 ssh 分布式运行速度有差异吗？
• 我记得你之前跑过 tensorflow 的多机实验（ssh 方式还是 k8s 方式），速度有印象吗？

[pineking]

@xuerq 一些结论看是否正确：

• mxnet 分布式速度在 k8s 中和 ssh 中没有区别， k8s 只负责任务的调度，而和 ssh 不同点在于 pod 之前的网络互连速度，目前 pod 之间通过 flannel 配置 host-gw 模式也是可以互通的

[xuerq]

没用单独测过 ssh 的分布式速度 对比上一帖子：ssh 方式 分布式2机*每机4卡的速度 ssh: 8010.492（张图像每秒） k8s: 7586.296786（张图像每秒） k8s 约是 ssh 速度的 94.7%

5% 的差异是不是来自 docker 的封装？ docker 应该会有 1% 左右的性能下降

[wangkuiyi]

这个5%的差异是不是来自pod之间的通信依赖的是物理网络之上的一层软件实现的overley network？

按说docker不应该引入什么性能差异吧。

[xuerq]

有可能，但也不好说，5% 差距可能是正常的性能波动，可能需要多做几轮对比测试验证一下