increased inference latency for quantized model

[ZongqiangZhang]

I hava just reproduced the classification on resnet50+imagenet. The accuracy is excellent!

But there is a significant increase in inference latency for quantized model. Test results on resnet + imagenet + tesla t4:

• test(model, test_loader) takes 143 seconds
• test(quantized_model, test_loader) takes 1442 seconds

Does anybody hit the same issue ?

[zlifd]

hi, i got the same issue with you, may i know how do you solve the problem? thanks

[ZongqiangZhang]

@zlifd not resolved yet

[zlifd]

@zlifd not resolved yet

Oh, sad to hear that. But I guess the problem is the fake quantization was used in Vitis AI Quantizer, the tensors are not really integers but are represented by floating numbers. Maybe you can take a look from the link below: https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Configure-the-Quantize-Strategy Thx

[ZongqiangZhang]

@zlifd thx for your comments