ofrimasad
commented
5 months ago Hi @ullsen .
Thank you for your kind words.
My main suspect here is NMS.
If you are exporting your model with NMS (The default behavior that can be overridden), then you are benchmarking the NMS along with the model.
The NMS is an iterative algorithm that starts with thresholding and sorting. When the model is better trained, more predictions will get higher confidence, and more predictions will cross the threshold and be included in the sorting part and the iterative part of the NMS.
To validate this assumption, you can export with the following flag:
model.export(... , postprocessing=False, ...)You can also try calibrating the following params of the export function: num_pre_nms_predictions: (int) Number of predictions to keep before NMS. nms_threshold: (float) NMS threshold for the exported model. confidence_threshold: (float) Confidence threshold for the exported model.
Hope that helps.
ullsen
shaydeci
💡 Your Question
Hello everybody and first of all thanks to the super_gradients team for the great work!
I have a theory question on the YOLO NAS object detection models. I am training on a custom data set and check the CPU inference time after training. I observed that longer model training leads to larger inference times, which surprises me a lot. I thought the main aspect of inference is the number of weights, which should be the same for a given model size? Instead, I see an increasing inference time the longer I train the model. Despite the different times, the model sizes remain more or less the same.
For training the models I used an identical setup (same input image size, model size, pre-processing pipeline, training hardware etc etc) with different number of epochs (I tried both, training from scratch and resuming a model by loading weights). I tried the medium and the large model with the same behaviour. After converting the models to onnx I see the same effect.
Let me know if you need more details and thanks for the help!
Versions
No response