YOLOv5s PQT/QAT - Getting 0% mAP in TensorRT int8 .engine models

[olibartfast]

Hi I'm working inside NGC container image nvcr.io/nvidia/pytorch:23.04-py3 using a GPU RTX 3090

• These are my steps:

• Cloning the YOLOv5 repository:

  git clone https://github.com/ultralytics/yolov5
  cd yolov5
  git checkout v7.0

• Download the pretrained weights (e.g., yolov5s.pt):

  wget https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.pt

  • Quantization Steps

  • Cloning the yolo_deepstream repository:

    git clone https://github.com/NVIDIA-AI-IOT/yolo_deepstream.git

  • Copy QAT Files:

    cp -r yolo_deepstream/yolov7_qat/* yolov5/

  • Navigate to YOLOv5 Directory:

    cd yolov5

  • Run QAT Training Script:

    python scripts/qat-yolov5.py quantize yolov5x.pt --ptq=ptq.pt --qat=qat.pt --eval-ptq --eval-origin --cocodir=/coco

  • Export the model to onnx:

    python scripts/qat-yolov5.py export qat.pt --size=640 --save=qat.onnx --dynamic

  • Convert the onnx model to tensort engine:

    trtexec --onnx=qat.onnx  --saveEngine=qat.engine --int8 --buildOnly --memPoolSize=workspace:1024MiB 

  • Validate the engine model:

    python val.py --weights ptq.engine --data data/coco.yaml

    I am getting the expected results validating ptq.onnx/ptq.pt/qat.onnx/qat.pt while mAP is 0% on *.engine files, what am I doing wrong in the setup and steps above? thanks

[olibartfast]

solved, see here NVIDIA-AI-IOT/cuDLA-samples#27