Open chinmayjog13 opened 1 year ago
I am getting the same issue. I might try reverting to a previous build just to see if it was working before.
I just tried the previous releases, they didn't work. Upon inspecting the code, it looks as if the patch may have never gotten deployed to release. Here and here
There should be a line that say class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 10, uint8_t_int8_t, ConvInteger);
Maybe @jchen351 knows what happened.
I'm having the same issue. Can I help get the patch applied?
Same issue here. Do we have a timeline for the patch?
On the similar issue #3130, there is comment that may temporarily solve the issue. But when we change weight_type=QuantType.QInt8
to QuantType.QUInt8
, the onnx quantized model seems to perform slower, as also mentioned in that issue. And that also happened to me.
Beside, I think these issues happen due to onnx does not support ConvInteger
for signed Int8
datatype, here.
In ONNX Runtime docs, in Method Selection subsection, they said that:
In general, it is recommended to use dynamic quantization for RNNs and transformer-based models, and static quantization for CNN models.
So I tried to follow the an end-to-end example given in the docs, and somehow it worked, the weights are int8
. Here are steps that I did:
CalibrationDataReader
by using some facial images.quantize_static()
I have the same problem. I am finding that onnxruntime does not support the ConvInteger layer unfortunately, that means dynamic quantization is not working in onnxruntime if initial model has CNNs inside. Very sad!
I found a workaround to solve this, by setting:
quantize_dynamic(input_model, output_model, weight_type=QuantType.QInt8, nodes_to_exclude=['/conv1/Conv'])
Here, nodes_to_exclude
should be the list of conv layer name in your model. You may find it in the error message when loading the model using InferenceSession
.
For example, for the error message mentioned in the title of this issue, it would be: 'Conv_0_quant'.
Another workaround is to exclude all operators causing the issue. For example:
In:
quantize_dynamic(input_model, output_model, weight_type=QuantType.QInt8, operators_to_quantize=['MatMul', 'Attention', 'LSTM', 'Gather', 'Transpose', 'EmbedLayerNormalization'])
"Conv"
was removed from operators_to_quantize
.
I'm also facing the same error tired solving this problem.
Same issue for segformer model
NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name '/segformer/encoder/patch_embeddings.0/proj/Conv_quant'
I just hit this issue too. I will skip quantizing Conv operators for now.
same error, it only works with QUint8 but not QInt8
I found a workaround to solve this, by setting:
quantize_dynamic(input_model, output_model, weight_type=QuantType.QInt8, nodes_to_exclude=['/conv1/Conv'])
Here,
nodes_to_exclude
should be the list of conv layer name in your model. You may find it in the error message when loading the model usingInferenceSession
. For example, for the error message mentioned in the title of this issue, it would be: 'Conv_0_quant'.
Hi, I tried this while trying to quantize a whisper model (seq2seq) (see here) however I am getting the following error
NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name '/conv1/Conv_quant'
I tried to use nodes_to_exclude
in my AutoQuantizationConfig
to exclude the node but the error is still the same
qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False, nodes_to_exclude=['/conv1/Conv_quant'])
Any help would be appreciated! 🙏
I had to better describe the operators to quantize and the following worked for me:
dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)
dqconfig.nodes_to_exclude = ['Conv_quant']
dqconfig.operators_to_quantize=['MatMul', 'Attention', 'LSTM', 'Gather', 'Transpose', 'EmbedLayerNormalization']
Hope this helps, Thomas
still only works with QUInt8 for me
@ogencoglu
werer you able to solve the issue. I am facing the same with quantization and also i have more inference time with ONNX can be seen here any help will be appreciated
@fazankabir No solution found so far.
I am able to do quantization with:
model_fp32 = 'model_Segformer.onnx'
model_quant = "model_dynamic_quant.onnx"
quantized_model = quantize_dynamic(model_fp32, model_quant, weight_type=QuantType.QUInt8)
instead of QuantType.QInt8
but while doing the inference it is taking even more time. do you also have the issue with more inference time after exporting the segformer to onnx @ogencoglu ?
I haven't tested that (onnx is a must for my case) but quantized onnx model has longer inference time than full precision one. So I ditched segformer all together. @Fazankabir
I am able to do quantization with:
model_fp32 = 'model_Segformer.onnx' model_quant = "model_dynamic_quant.onnx" quantized_model = quantize_dynamic(model_fp32, model_quant, weight_type=QuantType.QUInt8)
instead of QuantType.QInt8
but while doing the inference it is taking even more time. do you also have the issue with more inference time after exporting the segformer to onnx @ogencoglu ?
Yeh you right, a lot of operators in onnx are running in CPU instead of GPU after being quantized (maybe they haven't supported yet), so it might take more time to run some quantized model
Describe the issue
Following the documentation, I dynamically quantiized a resnet based model. The model is quantized and saved without error. However, when I try to create an inference session using the quantized model, the code crashes with the following error.
This is a duplicate of #12558 , which was closed a few months ago so I guess the support should have been added in onnxruntime, but I am still getting the same error.
To reproduce
fp32_model quantized_model
Urgency
Not very urgent, but not too low priority also.
Platform
Linux
OS Version
Ubuntu 20.04
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
1.14.1
ONNX Runtime API
Python
Architecture
X64
Execution Provider
Default CPU
Execution Provider Library Version
No response