Closed hanxuel closed 5 years ago
I am working with gpu cluster, cuda/9.0.176 cudnn/7.0 jpeg/9b libpng/1.6.27 python_gpu/3.6.4, any idea about what happens?
Hi hanxuel, its unclear exactly why your configuration is crashing, but it likely has to do with the fact that you're using different versions of CUDA/CUDNN (maybe Tensorflow?). The compilation instructions in tf_grouping_compile.sh are only guaranteed to work with the configuration stated in the README: Tensorflow 1.12, CUDA 9.0, CUDNN 7.4.1 on Ubuntu 16.04 LTS
Hi drethage, I manage to handle the problem. I think it's because tensorflow-gpu must be run on GPU, while I happen to make it run on CPU device.
Extracted 248 samples from 12 items in train set Extracted 837 samples from 21 items in val set
2019-03-06 16:24:42.091695: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Parameters in model: 10076067
Layers in model: (Name - Shape - # weights) abstraction/points_to_15cm/simplified_pointnet/1x1_conv_1/weights:0 - (1, 1, 4, 64) - 256 abstraction/points_to_15cm/simplified_pointnet/1x1_conv_2/weights:0 - (1, 1, 64, 96) - 6144 abstraction/points_to_15cm/skip_15cm/1x1x1_conv/weights:0 - (1, 1, 1, 96, 256) - 24576 abstraction/points_to_15cm/skip_45cm/3x3x3_conv/weights:0 - (3, 3, 3, 96, 256) - 663552 abstraction/15cm_to_30cm/3d_convolution/2x2x2_conv/weights:0 - (2, 2, 2, 96, 128) - 98304 abstraction/15cm_to_30cm/3d_convolution/1x1_conv_1/weights:0 - (1, 1, 1, 128, 192) - 24576 abstraction/15cm_to_30cm/skip_30cm/1x1x1_conv/weights:0 - (1, 1, 1, 192, 256) - 49152 abstraction/15cm_to_30cm/skip_90cm/3x3x3_conv/weights:0 - (3, 3, 3, 192, 256) - 1327104 abstraction/30cm_to_60cm/3d_convolution/2x2x2_conv/weights:0 - (2, 2, 2, 192, 256) - 393216 abstraction/30cm_to_60cm/3d_convolution/1x1x1_conv_1/weights:0 - (1, 1, 1, 256, 512) - 131072 abstraction/30cm_to_60cm/skip_60cm/1x1x1_conv_1/weights:0 - (1, 1, 1, 512, 256) - 131072 abstraction/30cm_to_60cm/skip_180cm/3x3x3_conv/weights:0 - (3, 3, 3, 512, 256) - 3538944 spatial_pool/skip_spatial_pool/1x1x1_conv/weights:0 - (1, 1, 1, 512, 256) - 131072 upsampling/60cm_to_30cm/2x2x2_deconv/weights:0 - (2, 2, 2, 256, 768) - 1572864 upsampling/60cm_to_30cm/1x1x1_conv_1/weights:0 - (1, 1, 1, 256, 256) - 65536 upsampling/60cm_to_30cm/1x1x1_conv_2/weights:0 - (1, 1, 1, 256, 192) - 49152 upsampling/30cm_to_15cm/2x2x2_deconv/weights:0 - (2, 2, 2, 192, 704) - 1081344 upsampling/30cm_to_15cm/1x1x1_conv_1/weights:0 - (1, 1, 1, 192, 128) - 24576 upsampling/30cm_to_15cm/1x1x1_conv_3/weights:0 - (1, 1, 1, 384, 128) - 49152 upsampling/15cm_to_5cm/final_deconv/weights:0 - (3, 3, 3, 64, 384) - 663552 upsampling/15cm_to_5cm/final_conv/weights:0 - (3, 3, 3, 64, 22) - 38016 Traceback (most recent call last): File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _run_fn self._extend_graph() File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1352, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'QueryBallPoint' with these attrs. Registered devices: [CPU,XLA_CPU], Registered kernels: device='GPU'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "main.py", line 31, in
main()
File "main.py", line 26, in main
training.train(cla.config)
File "/cluster/home/haliang/fully-convolutional-point-network/training.py", line 245, in train
sess.run([init_g, init_l], {is_training_pl: True})
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'QueryBallPoint' with these attrs. Registered devices: [CPU,XLA_CPU], Registered kernels:
device='GPU'
Caused by op 'abstraction/points_to_15cm/simplified_pointnet/QueryBallPoint', defined at: File "main.py", line 31, in
main()
File "main.py", line 26, in main
training.train(cla.config)
File "/cluster/home/haliang/fully-convolutional-point-network/training.py", line 209, in train
queue_batch_placeholders['input_features_pl'], is_training_pl, dataset.get_num_learnable_classes(), batch_normalization_decay)
File "/cluster/home/haliang/fully-convolutional-point-network/fcpn.py", line 217, in build_model
grouped_points_xyz_and_features = self.radius_search_and_group(pointnet_locations, self.get_pointnet_radius(self._config['model']['pointnet']['spacing']), self._config['model']['pointnet']['neighbors'], points_xyz, points_features)
File "/cluster/home/haliang/fully-convolutional-point-network/fcpn.py", line 174, in radius_search_and_group
pointindices, = tf_grouping.query_ball_point(radius, num_neighbors, points_xyz, centroids_xyz)
File "tf_grouping/tf_grouping.py", line 22, in query_ball_point
return grouping_module.query_ball_point(xyz1, xyz2, radius, nsample)
File "", line 55, in query_ball_point
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/cluster/home/haliang/.local/lib64/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'QueryBallPoint' with these attrs. Registered devices: [CPU,XLA_CPU], Registered kernels: device='GPU'