Conv2DCustomBackpropInputOp only supports NHWC.

[Kylinlin]

Hi, when I run training.py, I got such error message, any idea please. by the way, the tensorflow version is 1.4.0, not compiled.

<module 'depthmotionnet.datareader' from '/home/kylin/demon/python/depthmotionnet/datareader/init.py'> Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(32, 4608), dtype=float32, device=/device:GPU:0) nothing to restore. no checkpoint found. train evolution "0_flow1" ... start iteration 0 2018-05-17 19:37:59.029835: E tensorflow/core/common_runtime/executor.cc:643] Executor failed to create kernel. Invalid argument: Conv2DCustomBackpropInputOp only supports NHWC. [[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]] stop iteration 0 Traceback (most recent call last): File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call return fn(*args) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn status, run_metadata) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC. [[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "training.py", line 604, in tf.app.run() File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "training.py", line 598, in main trainer.coordinator().raise_requested_exception() File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 407, in raise_requested_exception six.reraise(*self._exc_info_to_raise) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 296, in stop_on_exception yield File "/home/kylin/Applications/tfutils/python/tfutils/simpletrainer.py", line 280, in mainloop values = self._session.run(ops, options=run_options, run_metadata=run_metadata) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run run_metadata_ptr) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run feed_dict_tensor, options, run_metadata) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run options, run_metadata) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC. [[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

Caused by op 'tower_0/netFlow1/refine4/upconv/conv2d_transpose', defined at: File "training.py", line 604, in tf.app.run() File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "training.py", line 193, in main kernel_regularizer=_kernel_regularizer, File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 231, in flow_block conv_params, File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 108, in _refine kwargs, File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1423, in conv2d_transpose return layer.apply(inputs) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 671, in apply return self.call(inputs, *args, *kwargs) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 575, in call outputs = self.call(inputs, args, **kwargs) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1289, in call data_format=utils.convert_data_format(self.data_format, ndim=4)) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1211, in conv2d_transpose name=name) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 779, in conv2d_backprop_input use_cudnn_on_gpu=use_cudnn_on_gpu, data_format=data_format, name=name) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op op_def=op_def) File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC. [[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]

[benjaminum]

It looks like tensorflow is mapping some ops to the CPU. Unfortunately, tensorflow still does not support NCHW for some ops. Maybe you can try to strictly place ops on the GPU. Reducing the batch size to save memory could also help to place everything on the GPU.

[TheCrazyT]

Having the same problem, but changing batch size does not change anything.

[xxoospring]

can't work on ubuntu16.04 and tf1.15