Default MaxPoolingOp only supports NHWC on device type CPU

[JudasDie]

Hi, I met a problem when I run the tracker. Could you kindly help me to fix it?

2020-04-24 19:58:16.552084: E tensorflow/core/common_runtime/executor.cc:642] Executor failed to create kernel. Invalid argument: Default MaxPoolingOp only supports NHWC on device type CPU
     [[{{node pool0/MaxPool}}]]
Traceback (most recent call last):
  File "/data/home/v-hongyuan/programs/pycharm-community-2019.1.3/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/data/home/v-hongyuan/zhipeng/SiamRCNN/SiamR-CNN/tracking/../../tensorpack/tensorpack/predict/base.py", line 39, in __call__
    output = self._do_call(dp)
  File "/data/home/v-hongyuan/zhipeng/SiamRCNN/SiamR-CNN/tracking/../../tensorpack/tensorpack/predict/base.py", line 131, in _do_call
    return self._callable(*dp)
  File "/data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1232, in _generic_run
    return self.run(fetches, feed_dict=feed_dict, **kwargs)
  File "/data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU
     [[node pool0/MaxPool (defined at /data/home/v-hongyuan/programs/anaconda3/envs/pysot/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

[pvoigtlaender]

Hi, currently running without GPU is not supported. It seems you're trying to run on CPU and MaxPooling with the memory layout we use is unfortunately not supported on CPU by TensorFlow.

[JudasDie]

Hi, currently running without GPU is not supported. It seems you're trying to run on CPU and MaxPooling with the memory layout we use is unfortunately not supported on CPU by TensorFlow.

In fact, I run the code with GPUs. I don't know why it happens. When I test on the other server, it works well. Does the CUDA/cudnn versions count for this? Could you kindly provide a script to set up conda environment in the code?

[pvoigtlaender]

I don't have a specific script for that. In general, make sure to install tensorflow-gpu and not just tensorflow and then you can try a small example to verify if the gpu is used. There are probably some resources for this on the tensorflow website.

[wangaixue]

i meet the same question! you should change your version of the tensorflow-gpu to your own GPU-version. my 2080ti tensorflow-gpu 1.14.0 python 3.6 @JudasDie

[ckvic3]

i meet the same problem . the reason is the cuda version don't support the tensorflow-gpu . so the gpu can't be used in tensorflow . for my environment , my cuda version is 10.1 ,but the tensorflow-gpu 1.15 need cuda 10.0. so i uninstall the tensorflow-gpu 1.15 ,then use conda to install tensorflow-gpu 1.15 ,the conda will automatic install the cudatookit 10.0. i hope this can help you. @JudasDie

[pvoigtlaender]

Closing due to inactivity. Feel free to reopen if needed

[krishnendudas1979]

@pvoigtlaender , I got the below issue. Can u pls check once and let me know, my issue is same as what is discussed in this thread? In that case, can u pls let me know what exactly I need to do?

Installed tensorflow version:- 2.4.1

D:\Anaconda3\python.exe D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py 2021-06-16 18:40:21.454701: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2021-06-16 18:40:21.455585: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-06-16 18:40:27.247967: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-06-16 18:40:27.250382: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found 2021-06-16 18:40:27.251931: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303) 2021-06-16 18:40:27.263144: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: BGHM25154-L 2021-06-16 18:40:27.264750: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: BGHM25154-L 2021-06-16 18:40:27.266531: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-06-16 18:40:27.270359: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set Total Params before compile: 3743280 Total Params after compile: 3743280 (96, 96, 3) (1, 3, 96, 96) 2021-06-16 18:42:27.123539: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) Traceback (most recent call last): File "D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py", line 177, in database["younes"] = img_to_encoding("images/younes.jpg", FRmodel) File "D:\Python_project\PY_worrks\AI-work\Keras_module\Tensor_Face_Recognition_with_keras_CNN_week4\fr_utils.py", line 226, in img_to_encoding embedding = model.predict_on_batch(x_train) File "D:\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1820, in predict_on_batch outputs = self.predict_function(iterator) File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in call result = self._call(*args, kwds) File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 894, in _call return self._concrete_stateful_fn._call_flat( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1918, in _call_flat return self._build_call_outputs(self._inference_function.call( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 555, in call outputs = execute.execute( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU [[node FaceRecoModel/max_pooling2d/MaxPool (defined at \Python_project\PY_worrks\AI-work\Keras_module\Tensor_Face_Recognition_with_keras_CNN_week4\fr_utils.py:226) ]] [Op:__inference_predict_function_3592]**

Function call stack: predict_function

Process finished with exit code 1

[pvoigtlaender]

Hi,

please follow closely the instructions, especially the version of tensorflow, Siam R-CNN is based on TensorFlow 1 and it looks like you're using TensorFlow 2, which is very likely not compatible. Please install tensorflow via pip3 install tensorflow-gpu==1.15 as indicated in the readme.

[krishnendudas1979]

thanks @pvoigtlaender for your suggestion. Let me try and will update you accordingly.

[krishnendudas1979]

@pvoigtlaender I installed the tensorflow-gpu. Now the installed versions are --

tensorflow 2.4.1 tensorflow-estimator 2.5.0 tensorflow-gpu 2.5.0

But still I am getting this error.. Any suggestion pls.

D:\Anaconda3\python.exe D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py 2021-06-17 17:09:57.935266: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2021-06-17 17:09:57.936427: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-06-17 17:10:03.666014: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found 2021-06-17 17:10:03.666980: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303) 2021-06-17 17:10:03.680086: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: BGHM25154-L 2021-06-17 17:10:03.681098: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: BGHM25154-L 2021-06-17 17:10:03.682349: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. Total Params before compile: 3743280 Total Params after compile: 3743280 (96, 96, 3) (1, 3, 96, 96) 2021-06-17 17:12:47.458009: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) Traceback (most recent call last): File "D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py", line 177, in database["younes"] = img_to_encoding("images/younes.jpg", FRmodel) File "D:\Python_project\PY_worrks\AI-work\Keras_module\Tensor_Face_Recognition_with_keras_CNN_week4\fr_utils.py", line 226, in img_to_encoding embedding = model.predict_on_batch(x_train) File "D:\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1916, in predict_on_batch outputs = self.predict_function(iterator) File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 889, in call result = self._call(*args, kwds) File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 956, in _call return self._concrete_stateful_fn._call_flat( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1960, in _call_flat return self._build_call_outputs(self._inference_function.call( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call outputs = execute.execute( File "D:\Anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU** [[node FaceRecoModel/max_pooling2d/MaxPool (defined at \Python_project\PY_worrks\AI-work\Keras_module\Tensor_Face_Recognition_with_keras_CNN_week4\fr_utils.py:226) ]] [Op:__inference_predict_function_3592]

Function call stack: predict_function

Process finished with exit code 1

[pvoigtlaender]

Hi,

you still have tensorflow 2 installed. Best first uninstall everything related to tensorflow, something like

pip uninstall tensorflow pip uninstall tensorflow-gpu

and then install exactly the right version: pip install tensorflow-gpu==1.15

[krishnendudas1979]

@pvoigtlaender It didn't solve the issue. These are the installed version.

tensorboard 1.15.0 tensorflow-estimator 1.15.1 tensorflow-gpu 1.15.0

D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\Scripts\python.exe D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py 2021-06-17 23:57:05.209439: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found 2021-06-17 23:57:05.209966: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. WARNING:tensorflow:From D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. Total Params before compile: 3743280 2021-06-17 23:59:28.225188: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found 2021-06-17 23:59:28.226402: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2021-06-17 23:59:28.245129: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: BGHM25154-L 2021-06-17 23:59:28.246705: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: BGHM25154-L 2021-06-17 23:59:28.248219: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Total Params after compile: 3743280 (96, 96, 3) (1, 3, 96, 96) 2021-06-17 23:59:43.604087: E tensorflow/core/common_runtime/executor.cc:642] Executor failed to create kernel. Invalid argument: Default MaxPoolingOp only supports NHWC on device type CPU [[{{node max_pooling2d/MaxPool}}]] Traceback (most recent call last): File "D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/Face_Recognition.py", line 177, in database["younes"] = img_to_encoding("images/younes.jpg", FRmodel) File "D:\Python_project\PY_worrks\AI-work\Keras_module\Tensor_Face_Recognition_with_keras_CNN_week4\fr_utils.py", line 226, in img_to_encoding embedding = model.predict_on_batch(x_train) File "D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 1156, in predict_on_batch outputs = self.predict_function(inputs) File "D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3473, in call self._make_callable(feed_arrays, feed_symbols, symbol_vals, session) File "D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3410, in _make_callable callable_fn = session._make_callable_from_options(callable_opts) File "D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\client\session.py", line 1505, in _make_callable_from_options return BaseSession._Callable(self, callable_options) File "D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\lib\site-packages\tensorflow_core\python\client\session.py", line 1460, in init session._session, options_ptr) tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU [[{{node max_pooling2d/MaxPool}}]]

Process finished with exit code 1

[pvoigtlaender]

I think your GPU might not be found. Please first make sure that TensorFlow successfully finds your GPU

[krishnendudas1979]

@pvoigtlaender You are right. I executed the below portion of code

`import tensorflow as tf from tensorflow.python.client import device_lib if tf.test.gpu_device_name(): print('Default GPU Device:{}'.format(tf.test.gpu_device_name())) else: print("Please install GPU version of TF")

print(device_lib.list_local_devices())`

The result is--

D:\Python_project\PY_worrks\AI-work_py_3_7_9\venv\Scripts\python.exe D:/Python_project/PY_worrks/AI-work/Keras_module/Tensor_Face_Recognition_with_keras_CNN_week4/test.py 2021-06-19 19:45:53.127404: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found 2021-06-19 19:45:53.127864: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-06-19 19:45:54.498168: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Please install GPU version of TF [name: "/device:CPU:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 1620725758349662474 ] 2021-06-19 19:45:54.503733: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found 2021-06-19 19:45:54.504236: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2021-06-19 19:45:54.511044: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: BGHM25154-L 2021-06-19 19:45:54.511513: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: BGHM25154-L

Process finished with exit code 0

My laptop don't have the extra graphics card. Have only inbuild intel graphics card. How can I make sure that tensor flow finds and uses the GPU?

[pvoigtlaender]

Unfortunately it's not possible to run Siam R-CNN with an intel grahics card. You need an NVIDIA card with high enough CUDA compute capability

[hrx000]

Hi, currently running without GPU is not supported. It seems you're trying to run on CPU and MaxPooling with the memory layout we use is unfortunately not supported on CPU by TensorFlow.

In fact, I run the code with GPUs. I don't know why it happens. When I test on the other server, it works well. Does the CUDA/cudnn versions count for this? Could you kindly provide a script to set up conda environment in the code?

Hii I am getting a similar error, Please tell me how to fix it. Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1378, in _do_call return fn(*args) File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1361, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1454, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU [[{{node max_pool}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/content/drive/MyDrive/hifi3dface/data_prepare/run_data_preparation.py", line 363, in app.run(main) File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 308, in run _run_main(main, args) File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/content/drive/MyDrive/hifi3dface/data_prepare/run_data_preparation.py", line 340, in main prepare_test_data_RGB(FLAGS.img_dir, FLAGS.out_dir) File "/content/drive/MyDrive/hifi3dface/data_prepare/run_data_preparation.py", line 235, in prepare_test_data_RGB detect_2Dlmk_all_imgs( File "/content/drive/MyDrive/hifi3dface/data_prepare/run_data_preparation.py", line 88, in detect_2Dlmk_all_imgs LMK2D_batch = detect_2D_landmark.detect_2Dlmk68( File "/content/drive/MyDrive/hifi3dface/data_prepare/detect_2D_landmark.py", line 265, in detect_2Dlmk68 heatmap = sess.run(outputs, {inputs: test_img}) File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 968, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1191, in _run results = self._do_run(handle, final_targets, final_fetches, File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1371, in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1397, in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'max_pool' defined at (most recent call last): Node: 'max_pool' Default MaxPoolingOp only supports NHWC on device type CPU [[{{node max_pool}}]]

Original stack trace for 'max_pool':

data prepare failed