Using single GPU other than GPU 0 makes system unreachable

[akshay-raj-dhamija]

Hello, I used the following code to run a simple MNIST example using a GPU on a remote machine

gpu_no=1
training_model = cnn.CNNModelHelper(order="NCHW",name="training_net",use_cudnn=True)
training_model.net.RunAllOnGPU(gpu_id=gpu_no, use_cudnn=True)
training_model.param_init_net.RunAllOnGPU(gpu_id=gpu_no, use_cudnn=True)
workspace.ResetWorkspace()
soft=AddLeNetModel(training_model)
AddTrainingOperators(training_model, soft)
workspace.RunNetOnce(training_model.param_init_net)
workspace.CreateNet(training_model.net,overwrite=True,input_blobs=['data','label'])

To run the network for different iterations I use

workspace.FeedBlob("data", data, device_option)
workspace.FeedBlob("label", label, device_option)
workspace.RunNet(training_model.net, num_iter=1)

the above code initially works fine on gpu_no 1 and then hangs the gpus, ultimately making the system unreachable. Please note: the same works well without any issues on gpu_no 0.

[Yangqing]

This seems to be some issue with the GPU installation. What does nvidia-smi show? Is there some other processes running on gpu number 1 when it freezes?

[akshay-raj-dhamija]

Thanks for your response. The only processes running on both the GPUs were just the caffe2 code. Below, is a screen shot of nvidia-smi just before the GPU's froze.