After changing the gcc version

[feitiandemiaomi]

When I install gcc 4.9，the old issue was solved, but I meet a new problem following:

gpu2@gpu2-PowerEdge-R730:~/OWFO/TF_Deformable_Net$ python ./faster_rcnn/demo.py --model tf_deformable_net/restore_output/Resnet50_iter_145000.ckpt I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally filename: /home/gpu2/OWFO/TF_Deformable_Net/lib/psroi_pooling_layer/psroi_pooling.so

---

/home/gpu2/OWFO/TF_Deformable_Net/lib/psroi_pooling_layer/psroi_pooling.so W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_UNKNOWN I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (gpu2-PowerEdge-R730): /proc/driver/nvidia/version does not exist Tensor("Placeholder:0", shape=(?, ?, ?, 3), dtype=float32) Tensor("pool1:0", shape=(?, ?, ?, 64), dtype=float32) Tensor("bn2a_branch1/batchnorm/add_1:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("bn2a_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("res2a_relu:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("bn2b_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("res2b_relu:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("bn2c_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("res2c_relu:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("bn3a_branch1/batchnorm/add_1:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("bn3a_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res3a_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("bn3b_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res3b_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("bn3c_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res3c_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("bn3d_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res3d_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("bn4a_branch1/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4a_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4a_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4b_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4b_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4c_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4c_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4d_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4d_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4e_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4e_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("bn4f_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4f_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("rpn_conv/3x3/Relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("rpn_cls_score/BiasAdd:0", shape=(?, ?, ?, 18), dtype=float32) Tensor("rpn_cls_prob:0", shape=(?, ?, ?, ?), dtype=float32) Tensor("Reshape_2:0", shape=(?, ?, ?, 18), dtype=float32) Tensor("rpn_bbox_pred/BiasAdd:0", shape=(?, ?, ?, 36), dtype=float32) Tensor("Placeholder_1:0", shape=(?, 3), dtype=float32) Tensor("res4f_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res4f_relu:0", shape=(?, ?, ?, 1024), dtype=float32) Tensor("res5a_branch2a_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res5a_branch2b_offset/BiasAdd:0", shape=(?, ?, ?, 72), dtype=float32) Tensor("transpose:0", shape=(?, 512, ?, ?), dtype=float32) Tensor("res5a_branch2b/weights/read:0", shape=(512, 512, 3, 3), dtype=float32) Tensor("transpose_1:0", shape=(?, 72, ?, ?), dtype=float32) Tensor("bn5a_branch1/batchnorm/add_1:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("bn5a_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("res5b_branch2a_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res5b_branch2b_offset/BiasAdd:0", shape=(?, ?, ?, 72), dtype=float32) Tensor("transpose_2:0", shape=(?, 512, ?, ?), dtype=float32) Tensor("res5b_branch2b/weights/read:0", shape=(512, 512, 3, 3), dtype=float32) Tensor("transpose_3:0", shape=(?, 72, ?, ?), dtype=float32) Tensor("res5a_relu:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("bn5b_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("res5c_branch2a_relu:0", shape=(?, ?, ?, 512), dtype=float32) Tensor("res5c_branch2b_offset/BiasAdd:0", shape=(?, ?, ?, 72), dtype=float32) Tensor("transpose_4:0", shape=(?, 512, ?, ?), dtype=float32) Tensor("res5c_branch2b/weights/read:0", shape=(512, 512, 3, 3), dtype=float32) Tensor("transpose_5:0", shape=(?, 72, ?, ?), dtype=float32) Tensor("res5b_relu:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("bn5c_branch2c/batchnorm/add_1:0", shape=(?, ?, ?, 2048), dtype=float32) Tensor("conv_new_1_relu:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("rois:0", shape=(?, 5), dtype=float32) Tensor("conv_new_1_relu:0", shape=(?, ?, ?, 256), dtype=float32) Tensor("rois:0", shape=(?, 5), dtype=float32) Tensor("offset_reshape:0", shape=(?, 2, 7, 7), dtype=float32) Tensor("fc_new_2/fc_new_2:0", shape=(?, 1024), dtype=float32) Tensor("fc_new_2/fc_new_2:0", shape=(?, 1024), dtype=float32) Loading network Resnet50_test... Traceback (most recent call last): File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1022, in _do_call return fn(*args) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1000, in _run_fn self._extend_graph() File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1049, in _extend_graph self._session, graph_def.SerializeToString(), status) File "/home/gpu2/anaconda3/lib/python3.6/contextlib.py", line 89, in exit next(self.gen) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status))

tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'DeformConvOp' with these attrs. Registered devices: [CPU], Registered kernels:

device='GPU'; T in [DT_DOUBLE] device='GPU'; T in [DT_FLOAT]

 [[Node: res5a_branch2b/DeformConvOp = DeformConvOp[T=DT_FLOAT, data_format="NCHW", deformable_group=4, num_groups=1, padding="SAME", rates=[1, 1, 2, 2], strides=[1, 1, 1, 1]](transpose, res5a_branch2b/weights/read, transpose_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./faster_rcnn/demo.py", line 136, in saver.restore(sess,ckpt) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1439, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 767, in run run_metadata_ptr) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 965, in _run feed_dict_string, options, run_metadata) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run target_list, options, run_metadata) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'DeformConvOp' with these attrs. Registered devices: [CPU], Registered kernels: device='GPU'; T in [DT_DOUBLE] device='GPU'; T in [DT_FLOAT]

 [[Node: res5a_branch2b/DeformConvOp = DeformConvOp[T=DT_FLOAT, data_format="NCHW", deformable_group=4, num_groups=1, padding="SAME", rates=[1, 1, 2, 2], strides=[1, 1, 1, 1]](transpose, res5a_branch2b/weights/read, transpose_1)]]

Caused by op 'res5a_branch2b/DeformConvOp', defined at: File "./faster_rcnn/demo.py", line 126, in net = get_network(args.demo_net) File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/factory.py", line 36, in get_network return Resnet50_test() File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/Resnet50_test.py", line 21, in init self.setup() File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/Resnet50_test.py", line 221, in setup .deform_conv(3, 3, 512, 1, 1, biased=False, rate=2, relu=False, num_deform_group=4, name='res5a_branch2b') File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/network.py", line 41, in layer_decorated layer_output = op(self, layer_input, *args, **kwargs) File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/network.py", line 192, in deform_conv dconv = trans2NHWC(dconvolve(data, kernel, offset)) File "/home/gpu2/OWFO/TF_Deformable_Net/lib/networks/network.py", line 182, in i, k, o, strides = [1, 1, s_h, s_w], rates=[1, 1, rate, rate], padding=padding, num_groups=num_groups, deformable_group=num_deform_group) File "", line 45, in deform_conv_op File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op op_def=op_def) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/gpu2/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1264, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'DeformConvOp' with these attrs. Registered devices: [CPU], Registered kernels: device='GPU'; T in [DT_DOUBLE] device='GPU'; T in [DT_FLOAT]

 [[Node: res5a_branch2b/DeformConvOp = DeformConvOp[T=DT_FLOAT, data_format="NCHW", deformable_group=4, num_groups=1, padding="SAME", rates=[1, 1, 2, 2], strides=[1, 1, 1, 1]](transpose, res5a_branch2b/weights/read, transpose_1)]]

It seems cuda has error , Do you have some advices?

[Zardinality]

It seems your GPU driver is not properly installed, do you have normal nvidia-smi output? Can you run other operation in gpu by manually set the device to GPU? Did you set CUDA_VISIBLE_DEVICES to some other value? Personally I don't think it's related to my code, what do you think?

[feitiandemiaomi]

Me too, now ,the output of nvidia-smi are "nvidia: command not found" I am wondering if I need to reinstall cuda , I guess it was the change of gcc leads to the cuda failure

[Zealoe]

@feitiandemiaomi have you fixed the problem？ i also met the problem.

[Zardinality]

@Zealoe Do you also encounter this issue after the reinstal of gcc? I am afraid that the GPU cuda stuff is based on the original gcc. You might want to install the original version gcc or reinstall cuda in current setting.