Closed JosephChenHub closed 3 years ago
Hello @JosephChenHub , good work and thanks for sharing this analysis!
answers to your question:
--sensitivity
in https://github.com/NVIDIA/NeMo/blob/main/examples/asr/quantization/speech_to_text_quant_infer.py, it is not generalized solution, but we see it works for networks like quartzNet, efficientNet.Hello @JosephChenHub , good work and thanks for sharing this analysis!
answers to your question:
- We have a code generator to fuse the pointwise operation since 7.0 release. On some platform this can be further fused with the neighbor conv/gemm.
- We will keep enhance the code generator for pointwise op works instead of adding specific kernels for new activations like silu. BTW, could you use relu to replace the silu, see #997, not sure if it also helps in your case.
- the QAT support is in the next major release (around 2 months). Currently we have polygraphy to debug accuracy issue, and we plan to add some developer guide on how to debug accuracy issue. Also there is some opensource code that might give us some intuition on how to find these sensitive layers, see
--sensitivity
in https://github.com/NVIDIA/NeMo/blob/main/examples/asr/quantization/speech_to_text_quant_infer.py, it is not generalized solution, but we see it works for networks like quartzNet, efficientNet.
OK, hope TensorRT more powerful !
@JosephChenHub @ttyio thanks for the YOLOv5 quantization analysis and response. One point I might add is that SiLU (sometimes referred to as Swish) is a very common activation layer used in various vision AI models.
In YOLOv5 we originally used nn.LeakyReLU(0.1) (inherited from Joseph Redmon's YOLOv3), and were able to realize about +1 AP gain when moving from nn.LeakyReLU(0.1) to nn.SiLU(), with little training or inference speed/memory penalties.
Considering the QAT problem overall the SiLU/LeakyReLU(0.1) compromise is unclear to me as to whether a nn.LeakyReLU(0.1) implementation may be more favorable when targeting TensorRT deployments given the above shortcomings described by @ttyio.
Disclaimer: not a TensorRT or QAT expert.
EDIT: to modify the YOLOv5 baseline activation function you would change L36 in Conv() module in yolov5/models/common.py. Naturally models must be retrained with this change, this is an a change that can be made at export on SiLU-trained models. https://github.com/ultralytics/yolov5/blob/ed2c74218d6d46605cc5fa68ce9bd6ece213abe4/models/common.py#L30-L43
@JosephChenHub , in case you are not aware of, besides min-max, entropy, there is another percentile-max calibrator you can play with, a 99.99%
percentile max sample is in
https://github.com/NVIDIA/TensorRT/blob/master/demo/BERT/helpers/calibrator.py#L102
I will close this issue, please reopen if you still have question, thanks!
@JosephChenHub , in case you are not aware of, besides min-max, entropy, there is another percentile-max calibrator you can play with, a
99.99%
percentile max sample is inhttps://github.com/NVIDIA/TensorRT/blob/master/demo/BERT/helpers/calibrator.py#L102
I will close this issue, please reopen if you still have question, thanks!
Were you able to get better INT8 mAP by setting a different value in the calibrator.py? BTW YoloV5 3.0 has hardswish, which suffers from the same issues.
Hi @ttyio It's been a year since you posted about QAT support. Any updates on when it will be released?
Description
When doing INT8 PTQ (post-training quantization), we notice that the dropout of the accuracy on image classification or semantic segmentation is rather small. However, for the detectors like yolov5, there exists a huge gap of mAP between FP16 model and INT8 model. The comparison results are listed as the following, e.g., for YOLOv5s, the mAP on COCO validation set decreases from 0.362 to 0.054 (or 0.316, 0.054 is the result of original onnx graph and 0.316 the result of custom plugin of
SiLU
).The huge gap between 0.362 and 0.054 makes the model unreliable, and we observed that the yolov5s model utilizes an activation function![image](https://user-images.githubusercontent.com/22444830/110743971-e634c500-8273-11eb-9494-8c0ebc3dfb7d.png)
SiLU
(SiLU(x) = x * sigmoid(x)) which may reduce the accuracy due to its nonlinearity.Hence, we manually write a plugin
![silu2](https://user-images.githubusercontent.com/22444830/110744141-2bf18d80-8274-11eb-89b1-a5bee5c10e47.png)
SiLU
to replace the origin operatorsigmoid
andmul
in the onnx graph, which looks likeAs a result, we can see from the above table that our detector keeps stable and there still exists a little gap between the FP16 models and the INT8 model (for yolov5s, the mAP/FPS improves from 0.054/464 to 0.316/327 due to our custom plugin). Furthermore, we do partial quantization (use top 5 sensitive conv. layers) and the mAP is 0.332, which is still lower than 0.362. So questions come that
Layer(PointWiseV2): PWN(Sigmoid_213, Mul_214), Tactic: 24, 345[Int8(128,40,40)] -> 347[Int8(128,40,40)]
, and how the TensorRT implements the INT8 operation on this layer (sigmoid + mul) ?SiLU
astemplate <> device forceinline float _silu_op(const float input) { return input / (static_cast(1) + expf(-input));
}
template <>
device forceinline half _silu_op(const half input) {
typedef half T;
return input / (static_cast(1) + hexp(-input));
}
template
global void _kernel(const T restrict inputs, T restrict outputs, const size_t n ) {
const size_t tid = threadIdx.x + blockIdx.x * blockDim.x;
if (tid < n) {
outputs[tid] = _silu_op(inputs[tid]);
}
}