Closed peiwenhuang27 closed 3 years ago
@peiwenhuang27 any issues with "InternalError: Missing 0-th output from node model/layer_1/Conv2D_eightbit_requantize " message are because core/common_runtime/mkl_layout_pass.cc doesn't rewrite graph correctly.
TF_ENABLE_MKL_NATIVE_FORMAT must be set always.
The most likely case in your side should be those variables don't take effect in c++ code.
Since I'm not sure if the issue is because of Colab, I tried running it in my local machine. I set the variables using
import os
os['TF_ENABLE_ONEDNN_OPTS'] = '1'
os['TF_ENABLE_MKL_NATIVE_FORMAT'] = '1' # also tried '0', not sure what its value should be
But the following error still occurs:
2021-07-19 18:36:45.607306: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From main.py:24: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
Traceback (most recent call last):
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
return fn(*args)
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1358, in _run_fn
self._extend_graph()
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1398, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'QuantizedMatMulWithBiasAndDequantize' used by {{node model/dense/Tensordot/MatMul_eightbit_requantize}} with these attrs: [input_quant_mode="MIN_FIRST", Toutput=DT_FLOAT, T1=DT_QUINT8, T2=DT_QINT8, Tbias=DT_QINT32, transpose_a=false, transpose_b=false]
Registered devices: [CPU]
Registered kernels:
<no registered kernels>
[[model/dense/Tensordot/MatMul_eightbit_requantize]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 26, in <module>
predictions = sess.run(output, {input_tensor_name: x[:64]}) # 64, 257, 60, 1
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 968, in run
run_metadata_ptr)
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1191, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run
run_metadata)
File "/Users/joannehuang/Documents/Work/quantization/TF2_inference/Intel-Tensorflow-Env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'QuantizedMatMulWithBiasAndDequantize' used by node model/dense/Tensordot/MatMul_eightbit_requantize (defined at main.py:24) with these attrs: [input_quant_mode="MIN_FIRST", Toutput=DT_FLOAT, T1=DT_QUINT8, T2=DT_QINT8, Tbias=DT_QINT32, transpose_a=false, transpose_b=false]
Registered devices: [CPU]
Registered kernels:
<no registered kernels>
[[model/dense/Tensordot/MatMul_eightbit_requantize]]
From the command line log (This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA), looks like the environment variables have been set successfully.
if possible, could you pls share us your pb file and evaluation script?
Due to some reasons, I cannot directly upload my files here. I have emailed the files to you, please check your inbox for them. Thank you so much, I truly appreciate it!
By the way, I found the following in the release notes for Intel-Tensorflow 2.5:
Only native layout format is supported (The environment variable TF_ENABLE_MKL_NATIVE_FORMAT will not have any effect) The oneDNN optimizations in official TensorFlow will not include int8 quantization (it will still be available in Intel Optimized TensorFlow). It will be available in later versions of official TensorFlow.
It looks like TF_ENABLE_MKL_NATIVE_FORMAT
does not affect the environment, and that all the int8 quantized operations are not supported yet in official Tensorflow. Could it be the reason for the error?
By the way, I found the following in the release notes for Intel-Tensorflow 2.5:
Only native layout format is supported (The environment variable TF_ENABLE_MKL_NATIVE_FORMAT will not have any effect) The oneDNN optimizations in official TensorFlow will not include int8 quantization (it will still be available in Intel Optimized TensorFlow). It will be available in later versions of official TensorFlow.
It looks like
TF_ENABLE_MKL_NATIVE_FORMAT
does not affect the environment, and that all the int8 quantized operations are not supported yet in official Tensorflow. Could it be the reason for the error?
Need to set the TF_ENABLE_MKL_NATIVE_FORMAT=0 for int8 model execution with intel-tensorflow 2.5.0
I see! Thanks, it works now with Intel-Tensorflow, I encountered the problem mainly because I wanted the model to be able to run in official Tensorflow without Intel-Tensorflow (for simplicity in further inference session that will be run in ML.Net)
From offical TensorFlow 2.6, intel optimizations have been upstreamed into offical tensorflow.
In the future, it will become default path on CPU version.
I have the same error with tensorflow2.7, tfserving2.7.0-gpu
Missing 0-th output from {{node model/din_attention_layer/StatefulPartitionedCall_1/StatefulPartitionedCall/dense/Tensordot/MatMul_eightbit_requantize
I have Fix it
I have Fix it How did you solve it? I met the same problem 请问你是如何解决的呢,我遇到了相同的问题。
I have Fix it
log is :
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Missing 0-th output from {{node x}} [[SpopFnInvocation]] (1) Internal: Missing 0-th output from {{node x}} [[SpopFnInvocation]] [[SpopFnInvocation/_1]] 0 successful operations. 0 derived errors ignored.
import os
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'
os.environ['TF_ENABLE_MKL_NATIVE_FORMAT'] = '1'
This fix my issue. Thanks
Case 1
Framework: Tensorflow 2.5.0, Intel-Tensorflow 2.5.0 Environment: Google Colab
I have a successfully quantized model that is to be run for inference without using LPOT API, so I wrote the following inference code:
When running the line
predictions = sess.run(output, {input_tensor_name: x})
:This error happens with or without
Intel-Tensorflow==2.5.0
installed, nor is it resolved whenos.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'
is set explicitly.On the other hand, when I run the same code in VS Code with
Python 3.6.8 64-bit base: Conda
, it returns the same error message as in Case 2.Case 2
Framework: Tensorflow 2.4.0, Intel-Tensorflow 2.4.0 Environment: Google Colab
This case works well and prints out the MSE loss of the predictions, but when I uninstall
Intel-Tensorflow 2.4.0
and run it with official Tensorflow, while running the same line in Case 1 (predictions = sess.run(output, {input_tensor_name: x})
):The error persists even with
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'
set explicitly.I believe both cases are caused by the same type of error, i.e. No OpKernel was registered to support Op ...
I was given to understand that with official
Tensorflow v2.5
installed and the environment variableTF_ENABLE_ONEDNN_OPTS=1
set (reference), the quantized model is supposed to run with oneDNN supported. But it doesn't seem to be the case in neither v2.4 nor v2.5.Not sure if this is the right place to post this issue, but I have nowhere else to report the problem as Intel-Tensorflow doesn't allow issue reporting and Tensorflow developers usually ignore issues dependent on other packages. Any hint is greatly appreciated, thank you.