glenn-jocher
commented
1 month ago @jackfaubshner hello! 😊
It's great to see your enthusiasm and detailed approach towards implementing quantization with YOLOv3-tiny. Quantization can indeed be a bit tricky, but you're on the right track. Here are some insights and suggestions that might help you improve your results:
Observations and Suggestions
-
mAP Drop:
- It's not uncommon to see a drop in mAP when introducing quantization, especially if the model is not fine-tuned post-quantization. Quantization introduces approximation errors which can affect the model's performance.
-
Quantization Aware Training (QAT):
- Since you've already tried Post Training Static Quantization (PTQ), you might want to explore Quantization Aware Training (QAT). QAT simulates quantization during the training process, allowing the model to adapt to the quantization noise. This often results in better performance compared to PTQ.
- You can follow the PyTorch QAT tutorial to get started.
-
Model Modifications:
- Ensure that all layers that can benefit from quantization are included. Sometimes, certain layers might not be quantized properly, leading to suboptimal performance.
- Verify that the quantization and dequantization layers are correctly placed. Typically, you want to quantize inputs as early as possible and dequantize outputs as late as possible.
-
Calibration Dataset:
- For PTQ, the choice of calibration dataset is crucial. Ensure that the calibration dataset is representative of the data the model will see during inference. This helps in better scaling and reduces quantization errors.
-
Inference Mode:
- Ensure that the model is set to evaluation mode (
model.eval()) during inference. This ensures that layers like BatchNorm are not updating their statistics during inference.
- Ensure that the model is set to evaluation mode (
Example Code for QAT
Here's a simplified example to get you started with QAT:
import torch
import torch.quantization
from ultralytics import YOLO
# Load your model
model = YOLO("yolov3-tiny.yaml")
# Fuse Conv, BN, and ReLU layers
model.fuse_model()
# Prepare the model for QAT
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.quantization.prepare_qat(model, inplace=True)
# Train the model with QAT
# Ensure to use a representative dataset for training
# Example:
# train_model(model, train_loader, epochs=10)
# Convert the model to a quantized version
torch.quantization.convert(model.eval(), inplace=True)
# Save the quantized model
torch.save(model.state_dict(), 'yolov3-tiny-qat.pth')
# Load and run inference
model.load_state_dict(torch.load('yolov3-tiny-qat.pth'))
model.eval()
results = model("https://ultralytics.com/images/bus.jpg")Additional Resources
Keep experimenting and iterating on your approach. Quantization is a powerful tool, and with the right setup, you can achieve significant improvements in model efficiency with minimal loss in accuracy.
Best of luck with your quantization efforts! If you have any more questions, feel free to ask. 😊
jackfaubshner
Search before asking
Question
Hello everyone! :)
My goal for the next few days is to use this repository and implement Post Training Static Quantization and Quantization Aware Training to compare the mAP and inference speed with that of the non-quantized model
To get started, I am using YOLOv3-tiny (416 × 416) as it takes way less time to train (about a day for me). Once the results with YOLOv3-tiny are acceptable, I can move on to the full YOLOv3 (416 × 416)
What have I done so far:
Train YOLOv3-tiny (416 × 416) from scratch using this repository (with no modifications to the code). The below command was used:
python3.8 -m torch.distributed.run --nproc_per_node 4 train.py --data coco.yaml --epochs 300 --weight '' --cfg yolov3-tiny.yaml --img 416 --batch-size 128This model has an mAP of 0.31, the original Darknet model has an mAP of 0.331. I believe a 0.02 mAP is loss is acceptable.
Use the modifications from another issue (https://github.com/ultralytics/yolov3/issues/1734) to add quantization and dequantization layers in the beginning and end of the model
yolo.py:
Why use this method? The person who used this method said it worked for him, so, might as well start with something that works.
Anyway, doing a print(model) gives me the following output:
Once again, I trained the model with the below command:
python3.8 -m torch.distributed.run --nproc_per_node 4 train.py --data coco.yaml --epochs 300 --weight '' --cfg yolov3-tiny.yaml --img 416 --batch-size 128This model got an mAP of 0.25It did not however change my model size, maybe the author of that issue made some other modifications
3.
Instead of adding the dequantization layer the way it was done in the above mentioned issue, I added the dequantization layer inside the YOLOv3-tiny YAML file. It looks as follows:
The dequantization layer had to be added before the Detect layer to avoid running into errors.
The mAP was 0.25 (same as the method used in the previous issue)
Current questions I have: I am unsure if this change in mAP by using the quantization and dequantization layers was supposed to happen or not. I did add those layers, but made no other modifications to the model and the inference was performed in float32 mode as well. Yet, the mAP dropped.
I am going to keep up with my attempts to use quantization and get better mAP. Any inputs are greatly appreciated, I am new to all of this
Additional
No response