Performance differences depending on the amount of data set

Hogushake commented 11 months ago

Search before asking

[X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

hello everyone! I divided FLIR's ADAS dataset into two datasets and trained each with YOLOV5s. one is a dataset with only thermal images (train: 11,042, valid: 2,498), and the other is a dataset thermal + rgb images (train: 21,061, val: 2,229).

training proceeded as follows. epoch 20, batch size 16

I checked each performance value using val.py, --data used the path of the first thermal dataset, and --weight used the weight of each training data.

The results are as follows:

1) thermal dataset

 Class     Images  Instances          P          R      mAP50   mAP50-95: 100%|██████████| 36/36 [00:17<00:00,  2.04
                   all       1144      16688      0.733      0.404      0.443      0.253
                person       1144       4470      0.811      0.634      0.731      0.385
               bicycle       1144        170      0.612      0.518      0.505      0.291
                   car       1144       7128      0.845      0.757      0.834       0.57
            motorcycle       1144         55      0.844        0.6      0.679      0.349
                   bus       1144        179      0.764      0.492      0.615       0.43
                 truck       1144         46      0.409      0.478      0.294      0.192
         traffic light       1144       2002      0.775      0.515      0.585      0.253
          fire hydrant       1144         94      0.801      0.319      0.422       0.22
             stop sign       1144       2472      0.732       0.47      0.548      0.307
            skateboard       1144          3          1          0          0          0
          refrigerator       1144          6          1          0    0.00973    0.00875
            hair drier       1144         63      0.207     0.0662     0.0889     0.0332
Speed: 0.5ms pre-process, 4.5ms inference, 1.8ms NMS per image at shape (32, 3, 640, 640)

2) rgb + thermal dataset

 Class     Images  Instances          P          R      mAP50   mAP50-95: 100%|██████████| 36/36 [00:18<00:00,  1.99
                   all       1144      16688      0.778      0.374       0.44      0.251
                person       1144       4470      0.845      0.607      0.731      0.381
               bicycle       1144        170      0.718      0.478      0.496      0.279
                   car       1144       7128      0.878      0.736      0.837      0.572
            motorcycle       1144         55      0.907      0.545      0.633      0.344
                   bus       1144        179      0.818      0.451      0.618      0.423
                 truck       1144         46      0.394      0.348      0.277      0.182
         traffic light       1144       2002      0.811      0.534      0.626      0.276
          fire hydrant       1144         94      0.772      0.298      0.341      0.177
             stop sign       1144       2472      0.774      0.456      0.548      0.308
            skateboard       1144          3          1          0     0.0585     0.0175
          refrigerator       1144          6          1          0      0.019     0.0139
            hair drier       1144         63      0.423     0.0317     0.0996     0.0346
Speed: 0.5ms pre-process, 4.5ms inference, 1.9ms NMS per image at shape (32, 3, 640, 640)

and there is not much difference in performance. I don't know why. Did I make an error??

Additional

No response

glenn-jocher commented 11 months ago

@Hogushake hi there!

Great job on dividing the FLIR's ADAS dataset and training two different YOLOv5 models. Regarding the performance difference between the thermal dataset and the RGB + thermal dataset, it's important to note that there can be various factors affecting the results.

Firstly, the model's performance heavily relies on the characteristics and diversity of the dataset. If the two datasets have similar distributions and contain enough representative samples, it is expected that the model's performance won't differ significantly. However, if the thermal dataset lacks certain object classes or contains imbalanced samples, this can affect the model's performance.

Additionally, the amount of training data plays a crucial role in the model's performance. Training with a larger dataset tends to improve the model's generalization ability and robustness. It's possible that the RGB + thermal dataset with more training examples has slightly better performance compared to the thermal-only dataset.

Lastly, the hyperparameter settings and training parameters such as learning rate, optimizer, and augmentation techniques can also influence the model's performance. Ensuring that the training settings are appropriate for the dataset is crucial for achieving optimal results.

In conclusion, considering the factors mentioned above such as dataset characteristics, data diversity, and training parameters, the slight performance difference between the two datasets is understandable. Make sure to analyze these factors and assess the quality of the data to understand the observed variation.

github-actions[bot] commented 10 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

ultralytics / yolov5