Wrong Confusion Matrix Results when Training Yolov5 with a custom Dataset

[IkrameBeggar]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello I trained yolov5s on a custom dataset with 50 epochs and batch size of 16 but after training the model I evaluated its performance on the test set and noticed that the mAP was 94% which was a bit weird for me. So I checked the confusion matrix and noticed that the results were wrong. Here are my training specifications:

1. dataset: The dataset was mainly published for pose estimation but I did some preprocessing on RoboFlow to make it suitable for object detection (I am not sure if this can be an issue or not). The data contains a single class and is divided to 25000 images for training, 4600 images for validation and 1198 images for testing.
2. Model Configuration: I have used yolov5s model configuration
3. Training Parameters: I have used the default training parameters except for the number of classes I set it to 1 and the activation function. Concerning the weights, I started the training from the petrained weightsof ultralytics (yolov5s.pt)
4. Activation Function: For the activation function I have changed the SiLU() to LeakyReLU(0.1015625, inplace=True). The reason I have done this is because my model will deployed later on an FPGA board and the SiLU activation function is not supported by the board.
5. Training Platform: I trained the model on my laptop with RTX4060 GPU with size 8GB

I hope you can help me fix this issue as I am confused what I can try to resolve the problem. Thank you very much in advance for guidance and support.

Additional

No response

[github-actions[bot]]

👋 Hello @IkrameBeggar, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@IkrameBeggar hello! Thanks for reaching out with your issue. It's great to hear that you've achieved a high mAP on your custom dataset, but I understand your concern about the confusion matrix results.

Here are a few steps you can take to troubleshoot the issue:

1. Data Integrity: Double-check your dataset annotations and class labels to ensure they are correct and consistent. Sometimes, preprocessing steps can introduce errors.

2. Activation Function: Changing the activation function can affect the model's performance. Although LeakyReLU is a valid alternative, it might require re-tuning other hyperparameters. Consider running a hyperparameter sweep to find the best combination for your modified architecture.

3. Model Evaluation: Make sure you're evaluating the model on the correct test set with the right command and parameters. Also, ensure that the test set is not leaking into the training or validation sets.

4. Visual Inspection: Visually inspect the predictions by using the --save-conf and --save-txt flags during detection to save the outputs and manually check a few examples.

5. Version Check: Verify that you're using the latest version of YOLOv5, as updates often include bug fixes and performance improvements.

6. Hyperparameters: Since you've changed the activation function, consider experimenting with different learning rates or other hyperparameters.

If after these checks the issue persists, please provide more details about the confusion matrix discrepancies you're observing. Are there specific classes being misclassified, or is it a general issue across all classes?

For further guidance, you can refer to our documentation at https://docs.ultralytics.com/yolov5/.

Keep up the good work, and I'm confident you'll get to the bottom of this! 🚀

[IkrameBeggar]

@IkrameBeggar hello! Thanks for reaching out with your issue. It's great to hear that you've achieved a high mAP on your custom dataset, but I understand your concern about the confusion matrix results.

Here are a few steps you can take to troubleshoot the issue:

1. Data Integrity: Double-check your dataset annotations and class labels to ensure they are correct and consistent. Sometimes, preprocessing steps can introduce errors.
2. Activation Function: Changing the activation function can affect the model's performance. Although LeakyReLU is a valid alternative, it might require re-tuning other hyperparameters. Consider running a hyperparameter sweep to find the best combination for your modified architecture.
3. Model Evaluation: Make sure you're evaluating the model on the correct test set with the right command and parameters. Also, ensure that the test set is not leaking into the training or validation sets.
4. Visual Inspection: Visually inspect the predictions by using the --save-conf and --save-txt flags during detection to save the outputs and manually check a few examples.
5. Version Check: Verify that you're using the latest version of YOLOv5, as updates often include bug fixes and performance improvements.
6. Hyperparameters: Since you've changed the activation function, consider experimenting with different learning rates or other hyperparameters.

If after these checks the issue persists, please provide more details about the confusion matrix discrepancies you're observing. Are there specific classes being misclassified, or is it a general issue across all classes?

For further guidance, you can refer to our documentation at https://docs.ultralytics.com/yolov5/.

Keep up the good work, and I'm confident you'll get to the bottom of this! 🚀

Hello again, I have another question concerning my results. I have tested my model (trained on a single class: craft) on coco dataset to see if the model overfitted during the training (since the mAP is very high) I noticed that the model detected objects as a craft (wrong detection). what can be the reason for this please? and what can I do to fix it? You can see below some examples of objects from coco dataset detected as a craft and some samples of the carfts.

[glenn-jocher]

@IkrameBeggar, it's not uncommon for a model trained on a specific dataset to incorrectly classify objects when tested on a completely different dataset, especially if the new dataset contains objects and backgrounds that were not present in the training data. This can be indicative of overfitting, where the model has learned to recognize the training data very well but fails to generalize to new, unseen data.

Here are some steps to address this:

1. Data Augmentation: Increase the diversity of your training data using data augmentation techniques. This helps the model generalize better to new environments.

2. Regularization: Apply regularization techniques such as dropout or weight decay to prevent the model from overfitting to the training data.

3. Cross-Dataset Evaluation: If you plan to use the model on a variety of datasets, consider training with a more diverse dataset that includes a wider range of objects and backgrounds.

4. Fine-Tuning: If you need the model to perform well on COCO or similar datasets, consider fine-tuning the model on a subset of COCO dataset with the 'craft' class included.

5. Review Training Data: Ensure that your training data is representative of the problem space and does not contain biases or artifacts that could lead the model to learn incorrect features.

6. Threshold Adjustment: Adjust the confidence threshold for detections to reduce false positives. This might require some trial and error to find the optimal threshold.

Remember, a high mAP on the training dataset is a good sign, but the ultimate goal is to have a model that performs well on the data it will encounter in its intended environment. Keep iterating and testing, and you'll improve the model's robustness and generalization. 🛠️

[IkrameBeggar]

@IkrameBeggar, it's not uncommon for a model trained on a specific dataset to incorrectly classify objects when tested on a completely different dataset, especially if the new dataset contains objects and backgrounds that were not present in the training data. This can be indicative of overfitting, where the model has learned to recognize the training data very well but fails to generalize to new, unseen data.

Here are some steps to address this:

1. Data Augmentation: Increase the diversity of your training data using data augmentation techniques. This helps the model generalize better to new environments.
2. Regularization: Apply regularization techniques such as dropout or weight decay to prevent the model from overfitting to the training data.
3. Cross-Dataset Evaluation: If you plan to use the model on a variety of datasets, consider training with a more diverse dataset that includes a wider range of objects and backgrounds.
4. Fine-Tuning: If you need the model to perform well on COCO or similar datasets, consider fine-tuning the model on a subset of COCO dataset with the 'craft' class included.
5. Review Training Data: Ensure that your training data is representative of the problem space and does not contain biases or artifacts that could lead the model to learn incorrect features.
6. Threshold Adjustment: Adjust the confidence threshold for detections to reduce false positives. This might require some trial and error to find the optimal threshold.

Remember, a high mAP on the training dataset is a good sign, but the ultimate goal is to have a model that performs well on the data it will encounter in its intended environment. Keep iterating and testing, and you'll improve the model's robustness and generalization. 🛠️

Hello again @glenn-jocher, thank for your guidance and help. I really appreciate it. can you please tell if this is the right way to add dropout layer? because I am having an error during the training. (ps: I modified `yolov5s.yaml)

backbone:
  # [from, number, module, args]
  [[-1, 1, Conv, [64, 6, 2, 2]],  # 0-P1/2
   [-1, 1, nn.Dropout, [0.1]],
   [-1, 1, Conv, [128, 3, 2]],  # 1-P2/4
   [-1, 1, nn.Dropout, [0.1]],
   [-1, 3, C3, [128]],
   [-1, 1, Conv, [256, 3, 2]],  # 3-P3/8
   [-1, 1, nn.Dropout, [0.1]],
   [-1, 6, C3, [256]],
   [-1, 1, Conv, [512, 3, 2]],  # 5-P4/16
   [-1, 1, nn.Dropout, [0.1]],
   [-1, 9, C3, [512]],
   [-1, 1, Conv, [1024, 3, 2]],  # 7-P5/32
   [-1, 1, nn.Dropout, [0.1]],
   [-1, 3, C3, [1024]],
   [-1, 1, SPPF, [1024, 5]],  # 9
  ]

[glenn-jocher]

Hello @IkrameBeggar, I'm glad to help!

The way you're trying to add the dropout layer in the yolov5s.yaml file isn't quite correct. In YOLOv5's architecture, layers are defined as a list of operations, and the dropout layer should be integrated within a module, not added directly into the backbone list.

Here's how you can correctly add a dropout layer within a custom module:

1. Define a new custom module that includes a convolutional layer followed by a dropout layer.
2. Replace the standard convolutional layer in the backbone with your custom module.

For example, you could define a custom module like this:

# Define a custom module with Convolution followed by Dropout
[[[-1, 1, Conv, [64, 6, 2, 2]],  # Convolution
  [-1, 1, nn.Dropout, [0.1]]],    # Dropout

Then, use this custom module in the backbone definition:

backbone:
  # [from, number, module, args]
  [[-1, 1, CustomConvDropout, [64, 6, 2, 2]],  # Custom module
   # ... rest of the backbone
  ]

Remember to define the CustomConvDropout module in the model's code, specifying the convolution and dropout operations.

However, please note that adding dropout directly to the backbone of YOLOv5 is not common practice and may not be supported directly by the architecture without modifying the source code. You would typically need to modify the model definition in the Python code to create a new module that includes both convolution and dropout.

If you're not familiar with PyTorch and YOLOv5's codebase, this could be a bit complex. I would recommend starting with simpler forms of regularization, like weight decay, which can be set directly in the training command without modifying the architecture.

Good luck with your model training! 🚀

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐