avolkov1
commented
6 years ago @ErlendFax My hunch is that something about the multigpu hardware configuration on your system is not setup right. Could you run the following command to share your GPUs layout: nvidia-smi topo -m and paste the output. Mine looks like this:
$ nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 mlx5_0 CPU Affinity
GPU0 X SOC SOC SOC SOC 0-15
GPU1 SOC X PHB PHB PHB 16-31
GPU2 SOC PHB X PIX PHB 16-31
GPU3 SOC PHB PIX X PHB 16-31
mlx5_0 SOC PHB PHB PHB X
Legend:
X = Self
SOC = Connection traversing PCIe as well as the SMP link between CPU sockets(e.g. QPI)
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
PIX = Connection traversing a single PCIe switch
NV# = Connection traversing a bonded set of # NVLinksTry with single GPU on each GPU separately and verify it works fine for other GPUs:
CUDA_VISIBLE_DEVICES=1 python mnist_tfrecord_mgpu.py
CUDA_VISIBLE_DEVICES=2 python mnist_tfrecord_mgpu.py
CUDA_VISIBLE_DEVICES=3 python mnist_tfrecord_mgpu.pyTry using 2 GPUs and vary which pairs you're using. You should observe some speedup:
CUDA_VISIBLE_DEVICES=0,1 python mnist_tfrecord_mgpu.py
CUDA_VISIBLE_DEVICES=0,2 python mnist_tfrecord_mgpu.py
CUDA_VISIBLE_DEVICES=1,2 python mnist_tfrecord_mgpu.pyI'm running on Tesla GPUs and I don't have access to a workstation setup with multi GeForce GPUs so it's hard for me to test. But refer to this link for an example and issues with GeForce setup:
https://github.com/rossumai/keras-multi-gpu/blob/master/blog/docs/hardware.md
Maybe @bzamecnik could offer some advise b/c he also ran these on GeForce multigpu workstations. Also, if you can post results of CUDA SDK utilties:
bandwidthTest - pure unidirectional bandwidth on single GPU
p2pBandwidthLatencyTest - combination of uni/bi-direction communication between pairsI'm using docker container: tensorflow/tensorflow:1.3.0-devel-gpu via nvidia-docker. My results on P40 4 GPU machine:
# CUDA_VISIBLE_DEVICES=0 python ./examples/mnist/mnist_tfrecord_mgpu.py
Epoch 1/5
429/429 [==============================] - 5s - loss: 0.2050 - acc: 0.9378
Epoch 2/5
429/429 [==============================] - 3s - loss: 0.0768 - acc: 0.9773
Epoch 3/5
429/429 [==============================] - 3s - loss: 0.0629 - acc: 0.9820
Epoch 4/5
429/429 [==============================] - 3s - loss: 0.0528 - acc: 0.9845
Epoch 5/5
429/429 [==============================] - 3s - loss: 0.0477 - acc: 0.9863
[TRAINING] finished in 19975 ms# CUDA_VISIBLE_DEVICES=0,1 python ./examples/mnist/mnist_tfrecord_mgpu.py
Epoch 1/5
214/214 [==============================] - 3s - loss: 0.2845 - acc: 0.9157
Epoch 2/5
214/214 [==============================] - 2s - loss: 0.0795 - acc: 0.9761
Epoch 3/5
214/214 [==============================] - 2s - loss: 0.0601 - acc: 0.9819
Epoch 4/5
214/214 [==============================] - 2s - loss: 0.0523 - acc: 0.9846
Epoch 5/5
214/214 [==============================] - 2s - loss: 0.0475 - acc: 0.9863
[TRAINING] finished in 12697 ms# CUDA_VISIBLE_DEVICES=0,1,2,3 python ./examples/mnist/mnist_tfrecord_mgpu.py
Epoch 1/5
107/107 [==============================] - 4s - loss: 0.5218 - acc: 0.8619
Epoch 2/5
107/107 [==============================] - 1s - loss: 0.1063 - acc: 0.9684
Epoch 3/5
107/107 [==============================] - 1s - loss: 0.0822 - acc: 0.9758
Epoch 4/5
107/107 [==============================] - 1s - loss: 0.0697 - acc: 0.9795
Epoch 5/5
107/107 [==============================] - 1s - loss: 0.0577 - acc: 0.9829
[TRAINING] finished in 10457 ms
bzamecnik
ErlendFax
majiali1995
I'm having issues using multiple GPUs. When I use
CUDA_VISIBLE_DEVICES=0 python mnist_tfrecord_mgpu.pyeverything works fine. When I useCUDA_VISIBLE_DEVICES=0,1,2,3 python mnist_tfrecord_mgpu.pyeverything seems normal, no printed errors, but training time has increased dramatically, and loss and accuracy barely change (if changed at all).TensorFlow 1.3.0
Keras 2.0.8
Cuda 8.0.61
cuDNN 6.0.21
Anyone else experiencing similar behavior? Looks like it's not even using all four GPUs.
Console output when training with four GPUs:
(...)
nvidia-smi: