search

Linaom1214 / TensorRT-For-YOLO-Series

tensorrt for yolo series (YOLOv11,YOLOv10,YOLOv9,YOLOv8,YOLOv7,YOLOv6,YOLOX,YOLOv5), nms plugin support
942 stars 159 forks source link

Getting an issue with int8 precession #45

Closed PavanAchari closed 2 years ago

PavanAchari commented 2 years ago

Hi First of all Thankyou very much for sharing this with us I am getting an issue when I am trying to convert the Yolov6 model weights to Tensorrt with int8 precession the conversion has been done but the detections are not working and also I have tried to reduce the threshold to 0.001 still the detections are not working and when I convert that model with fp32,fp16 is working fine Could you please help me with this

Thanks & Regards, Pavan

Linaom1214 commented 2 years ago

Use the coco dataset?

PavanAchari commented 2 years ago

It's a custom dataset I have used

xiaohui0225 commented 2 years ago

I have the same problem， too. And I used COCO datasets as the calibration dataset. And tested different "--calib_num_images". No matter how hard I try, the detections are not working.

Linaom1214 commented 2 years ago

I have the same problem， too. And I used COCO datasets as the calibration dataset. And tested different "--calib_num_images". No matter how hard I try, the detections are not working.

Thanks for your feedback, I will try to troubleshoot this as soon as possible

Linaom1214 commented 2 years ago

I have the same problem， too. And I used COCO datasets as the calibration dataset. And tested different "--calib_num_images". No matter how hard I try, the detections are not working.

which model?

Linaom1214 commented 2 years ago

@xiaohui0225 @PavanAchari

https://github.com/Linaom1214/TensorRT-For-YOLO-Series/blob/8c9189ef6d50ea1c36b1232b172f3221847e5791/image_batch.py#L154

change the code 

return  image / 255. , scale