search

DerryHub / BEVFormer_tensorrt

BEVFormer inference on TensorRT, including INT8 Quantization and Custom TensorRT Plugins (float/half/half2/int8).
Apache License 2.0
430 stars 71 forks source link

'CustomMSDeformableAttentionTRTP' gives incorrect outputs in Pytorch inference #84

Closed Jian-danai closed 1 year ago

Jian-danai commented 1 year ago

Hi,

It looks CustomMSDeformableAttentionTRTPgives incorrect outputs in Pytorch inference (different from CustomMSDeformableAttentionTRTand CustomMSDeformableAttention), although the Pytorch inference result is not that important.

The reasons caused the difference are: shapes: reference_points and attention_weights are reshaped in TRTP, specifically, the shapes of tensors fed into the forward funtion are different softmax: There is no softmax for attention_weights in CustomMSDeformableAttentionTRTP

Are there any reasons to give different outputs in TRTP? Or it is just a bug for Pytorch inference but does not matter? Thanks.

DerryHub commented 1 year ago

Softmax has been merged in OP MultiScaleDeformableAttnTRT of CustomMSDeformableAttentionTRTP for better quantization.

Jian-danai commented 1 year ago

Thanks. I tested the latest code it looks TRTP pytorch inference gives normal outputs, I will close this issue.