Closed jackonealll closed 2 years ago
Hi, I think your custom dataset may be too small so that it is overfitting. You can early stop the training when the accuracy does not increase for several epochs.
My dataset has this setup: train (6k images), valid (200 images) and test (700 images).
I will try to increase the data augmentation on train dataset.
Thank you!
@tangjiuqi097 I am experiencing strange behavior.
As suggested I applied another data augmentation. Now I have 27k images in the train dataset.
Before I had the ratio of 1:4, now 1:20.
But, before I got a AP@0.50 of 90%. Now, after the data augmentation my AP@0.50 is 83%.
Is it possible that more images can negatively influence in the CNN training?
Below are properties used in data augmentation:
data_gen_args = dict(
brightness_range=[0.4, 1.5],
rotation_range=20,
zoom_range=0.2,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
horizontal_flip=True,
vertical_flip=True,
fill_mode='constant',
zca_whitening=True,
cval=0
)
More images should have better performance, but some data augmentations may not be suitable.
@tangjiuqi097 Thank you again!
I have one more question. Could you please explain the purpose of line #94 of engine.py file?
I am asking this because I made some tests changing this line and I got stranges results. In short decreasing the IOU I am getting good results. But I think it would make my results worse.
IoU 0.50:0.95: bbox (ep 49): Line 94 commented
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.587
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.904
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.644
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.435
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.628
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.604
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.702
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.763
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.567
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.812
IoU 0.00:0.75: bbox (ep 49): Line 94 uncommented
Average Precision (AP) @[ IoU=0.00:0.75 | area= all | maxDets=100 ] = 0.872
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.00:0.75 | area= small | maxDets=100 ] = 0.250
Average Precision (AP) @[ IoU=0.00:0.75 | area=medium | maxDets=100 ] = 0.766
Average Precision (AP) @[ IoU=0.00:0.75 | area= large | maxDets=100 ] = 0.902
Average Recall (AR) @[ IoU=0.00:0.75 | area= all | maxDets= 1 ] = 0.853
Average Recall (AR) @[ IoU=0.00:0.75 | area= all | maxDets= 10 ] = 0.925
Average Recall (AR) @[ IoU=0.00:0.75 | area= all | maxDets=100 ] = 0.952
Average Recall (AR) @[ IoU=0.00:0.75 | area= small | maxDets=100 ] = 0.250
Average Recall (AR) @[ IoU=0.00:0.75 | area=medium | maxDets=100 ] = 0.866
Average Recall (AR) @[ IoU=0.00:0.75 | area= large | maxDets=100 ] = 0.976
IoU 0.5: bbox (ep 49): Line 94 uncommented [0.5]
Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.904
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.867
Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.922
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 1 ] = 0.870
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 10 ] = 0.944
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.971
Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.904
Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.992
IoU 0.4: bbox (ep 49): Line 94 uncommented [0.4]
Average Precision (AP) @[ IoU=0.30:0.30 | area= all | maxDets=100 ] = 0.943
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.30:0.30 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.30:0.30 | area=medium | maxDets=100 ] = 0.892
Average Precision (AP) @[ IoU=0.30:0.30 | area= large | maxDets=100 ] = 0.961
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets= 1 ] = 0.909
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets= 10 ] = 0.971
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets=100 ] = 0.978
Average Recall (AR) @[ IoU=0.30:0.30 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.30:0.30 | area=medium | maxDets=100 ] = 0.921
Average Recall (AR) @[ IoU=0.30:0.30 | area= large | maxDets=100 ] = 0.998
IoU 0.3: bbox (ep 49): Line 94 uncommented [0.3]
Average Precision (AP) @[ IoU=0.30:0.30 | area= all | maxDets=100 ] = 0.943
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.30:0.30 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.30:0.30 | area=medium | maxDets=100 ] = 0.892
Average Precision (AP) @[ IoU=0.30:0.30 | area= large | maxDets=100 ] = 0.961
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets= 1 ] = 0.909
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets= 10 ] = 0.971
Average Recall (AR) @[ IoU=0.30:0.30 | area= all | maxDets=100 ] = 0.978
Average Recall (AR) @[ IoU=0.30:0.30 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.30:0.30 | area=medium | maxDets=100 ] = 0.921
Average Recall (AR) @[ IoU=0.30:0.30 | area= large | maxDets=100 ] = 0.998
IoU 0.2: bbox (ep 49): Line 94 uncommented [0.2]
Average Precision (AP) @[ IoU=0.20:0.20 | area= all | maxDets=100 ] = 0.956
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.20:0.20 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.20:0.20 | area=medium | maxDets=100 ] = 0.894
Average Precision (AP) @[ IoU=0.20:0.20 | area= large | maxDets=100 ] = 0.978
Average Recall (AR) @[ IoU=0.20:0.20 | area= all | maxDets= 1 ] = 0.920
Average Recall (AR) @[ IoU=0.20:0.20 | area= all | maxDets= 10 ] = 0.972
Average Recall (AR) @[ IoU=0.20:0.20 | area= all | maxDets=100 ] = 0.981
Average Recall (AR) @[ IoU=0.20:0.20 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.20:0.20 | area=medium | maxDets=100 ] = 0.939
Average Recall (AR) @[ IoU=0.20:0.20 | area= large | maxDets=100 ] = 0.998
IoU 0.1: bbox (ep 49): Line 94 uncommented [0.1]
Average Precision (AP) @[ IoU=0.10:0.10 | area= all | maxDets=100 ] = 0.963
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.10:0.10 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.10:0.10 | area=medium | maxDets=100 ] = 0.897
Average Precision (AP) @[ IoU=0.10:0.10 | area= large | maxDets=100 ] = 0.987
Average Recall (AR) @[ IoU=0.10:0.10 | area= all | maxDets= 1 ] = 0.930
Average Recall (AR) @[ IoU=0.10:0.10 | area= all | maxDets= 10 ] = 0.977
Average Recall (AR) @[ IoU=0.10:0.10 | area= all | maxDets=100 ] = 0.983
Average Recall (AR) @[ IoU=0.10:0.10 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.10:0.10 | area=medium | maxDets=100 ] = 0.947
Average Recall (AR) @[ IoU=0.10:0.10 | area= large | maxDets=100 ] = 0.998
IoU 0.0: bbox (ep 49): Line 94 uncommented [0.0] (impossible!!!)
Average Precision (AP) @[ IoU=0.00:0.00 | area= all | maxDets=100 ] = 0.976
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.00:0.00 | area= small | maxDets=100 ] = 1.000
Average Precision (AP) @[ IoU=0.00:0.00 | area=medium | maxDets=100 ] = 0.950
Average Precision (AP) @[ IoU=0.00:0.00 | area= large | maxDets=100 ] = 0.989
Average Recall (AR) @[ IoU=0.00:0.00 | area= all | maxDets= 1 ] = 0.946
Average Recall (AR) @[ IoU=0.00:0.00 | area= all | maxDets= 10 ] = 0.998
Average Recall (AR) @[ IoU=0.00:0.00 | area= all | maxDets=100 ] = 0.998
Average Recall (AR) @[ IoU=0.00:0.00 | area= small | maxDets=100 ] = 1.000
Average Recall (AR) @[ IoU=0.00:0.00 | area=medium | maxDets=100 ] = 1.000
Average Recall (AR) @[ IoU=0.00:0.00 | area= large | maxDets=100 ] = 0.998
@jackonealll hi, If you use a smaller IoU threshold, the low quality prediction can be treated as true positive, so the performance can be better. It can be used to find out more results for different IoU threshold. The default metric is to evaluate the performance for the IoU threshold in 0.5:0.95, and of course we should remove this line to obey the default coco metric.
Hi @tangjiuqi097
I understand your point. But I think if I use a smaller IoU, more false positives can be found and impact negativly in the other metrics like Precision and F1-Score. In my case the results are getting better.
Bellow are other results with all metrics when changing the IoU value.
Also, I agree with you that we should use IoU threshold in 0.5:0.95. But in my experiments I need use a specific IoU like 0.5 this is the reason why I trying to understande this code to use the correct way.
if I use a smaller IoU, more false positives can be found
This IoU threshold is to evaluate the prediction, but not that for label assignment.
Maybe what you say is the confidence score threshold.
This issue is not active for a long time and it will be closed in 5 days. Feel free to re-open it if you have further concerns.
Hi,
I am studing your shared code using a custom dataset.
Here are key points:
I would like to understand why after 50 epochs the performance of my train starts get worse (overfitting?).
Here are the graph:
Run a validation code under my test dataset I got the following results using the saved weights:
Why the results on epoch 49 is better than results on epoch 99? Is there some option to save only the best weights?