ckpt_best.pth performing the worst

[cameron388]

💡 Your Question

I've trained a model to 100e on my own dataset.

When testing the model using "prediction = model.predict(processed_image_path, conf=confidence_threshold, fp16=False) #prediction = model.predict(processed_image_path, conf=confidence_threshold)" Im finding reproducibly (across different models) that ckpt_best.pth performs significantly worse in terms of recall and specificity.

For example here are some values running _best.pth compared to _latest.pth

_best

Confidence Threshold: 0.50 Precision: 0.8250 Recall: 0.7500 Specificity: 0.7941

Confidence Threshold: 0.70 Precision: 0.9355 Recall: 0.6591 Specificity: 0.9412

Confidence Threshold: 0.75 Precision: 0.9259 Recall: 0.5682 Specificity: 0.9412

Confidence Threshold: 0.80 Precision: 1.0000 Recall: 0.4773 Specificity: 1.0000

Confidence Threshold: 0.85 Precision: 1.0000 Recall: 0.1818 Specificity: 1.0000

_latest

Confidence Threshold: 0.50 Precision: 0.9649 Recall: 0.9910 Specificity: 0.9529

Confidence Threshold: 0.70 Precision: 0.9808 Recall: 0.9189 Specificity: 0.9765

Confidence Threshold: 0.80 Precision: 1.0000 Recall: 0.8288 Specificity: 1.0000

Confidence Threshold: 0.85 Precision: 1.0000 Recall: 0.6937 Specificity: 1.0000

Obviously this is a very surprising result so I'm wondering if something has gone wrong?

Versions

No response

[yevhen-k]

I have the same issue. I have yolo nas training set up on coco dataset. I trained model for a few epochs to test training set up. Then, I continued training from the checkpoint another 200 epochs. And it seems like the ckpt_best.pth wasn't updated since. Only average and latest checkpoints were updated.

[yevhen-k]

I use super-gradients==3.7.1. According to logs, ckpt_best.pth indeed was saved only once after the end of the first epoch of the first training.

Inspecting model checkpoints I've found the following:

best_model        acc: 0.0001     epochs: 1
average_model    acc: 0.3328     epochs: 200
latest_model     acc: 0.3328     epochs: 200

I have the same issue. I have yolo nas training set up on coco dataset. I trained model for a few epochs to test training set up. Then, I continued training from the checkpoint another 200 epochs. And it seems like the ckpt_best.pth wasn't updated since. Only average and latest checkpoints were updated.