Worse confidence scores on small Objects detect

PushpakBhoge commented 2 years ago

Search before asking

[X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi

I am trying to train a model on custom dataset, the dataset has only one class and very small objects. To give you an idea at any time on a image of 1200x1200 highest object size is 64x64 most of the boxes are below 32x32

I have tiled my dataset with patch size of 512x512 using SAHI crop I did this to preserve the resolution which is necessary for small Objects

The dataset is huge around 100k images. after tiling there are 75% images with atleast one object and 25% negative samples. I am also confident about the quality of annotations error rate in annotations is less than 5%

I have trained it with different set of params and long schedules the mAP is good, bounding boxes are also on spot most of the times

Problem I am facing is all the predictions have very low confidence score I am talking about 0.05 hardly any bbox has score higher than 0.25, anything higher than 0.1 is rare

in training as well I evolved params over 60 generations on a small subset of data to get best hyper parameters, Object loss is not decreasing. At epoch 1 bbox loss was 0.13 and object loss was 0.10 after 50 iterations bbox loss went down to 0.068 and mAP0.5 went up to 28. I have used pretrained v6 weights as a starting point, haven't tried training from scratch.q

But the object loss didn't budge, lowest I have seen is 0.998 that's it

What could be going wrong here ?

Additional

No response

github-actions[bot] commented 2 years ago

👋 Hello @PushpakBhoge, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Google Colab and Kaggle notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

glenn-jocher commented 2 years ago

@PushpakBhoge 👋 Hello! Thanks for asking about improving YOLOv5 🚀 training results. Low confidence scores are indicative of too short trainings and can be resolved by training for more epochs.

Most of the time good results can be obtained with no changes to the models or training settings, provided your dataset is sufficiently large and well labelled. If at first you don't get good results, there are steps you might be able to take to improve, but we always recommend users first train with all default settings before considering any changes. This helps establish a performance baseline and spot areas for improvement.

If you have questions about your training results we recommend you provide the maximum amount of information possible if you expect a helpful response, including results plots (train losses, val losses, P, R, mAP), PR curve, confusion matrix, training mosaics, test results and dataset statistics images such as labels.png. All of these are located in your project/name directory, typically yolov5/runs/train/exp.

We've put together a full guide for users looking to get the best results on their YOLOv5 trainings below.

Dataset

Images per class. ≥ 1500 images per class recommended
Instances per class. ≥ 10000 instances (labeled objects) per class recommended
Image variety. Must be representative of deployed environment. For real-world use cases we recommend images from different times of day, different seasons, different weather, different lighting, different angles, different sources (scraped online, collected locally, different cameras) etc.
Label consistency. All instances of all classes in all images must be labelled. Partial labelling will not work.
Label accuracy. Labels must closely enclose each object. No space should exist between an object and it's bounding box. No objects should be missing a label.
Label verification. View train_batch*.jpg on train start to verify your labels appear correct, i.e. see example mosaic.
Background images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.

Model Selection

Larger models like YOLOv5x and YOLOv5x6 will produce better results in nearly all cases, but have more parameters, require more CUDA memory to train, and are slower to run. For mobile deployments we recommend YOLOv5s/m, for cloud deployments we recommend YOLOv5l/x. See our README table for a full comparison of all models.

YOLOv5 Models

Start from Pretrained weights. Recommended for small to medium sized datasets (i.e. VOC, VisDrone, GlobalWheat). Pass the name of the model to the --weights argument. Models download automatically from the latest YOLOv5 release.

python train.py --data custom.yaml --weights yolov5s.pt
                                         yolov5m.pt
                                         yolov5l.pt
                                         yolov5x.pt
                                         custom_pretrained.pt

Start from Scratch. Recommended for large datasets (i.e. COCO, Objects365, OIv6). Pass the model architecture yaml you are interested in, along with an empty --weights '' argument:

python train.py --data custom.yaml --weights '' --cfg yolov5s.yaml
                                                  yolov5m.yaml
                                                  yolov5l.yaml
                                                  yolov5x.yaml

Training Settings

Before modifying anything, first train with default settings to establish a performance baseline. A full list of train.py settings can be found in the train.py argparser.

Epochs. Start with 300 epochs. If this overfits early then you can reduce epochs. If overfitting does not occur after 300 epochs, train longer, i.e. 600, 1200 etc epochs.
Image size. COCO trains at native resolution of --img 640, though due to the high amount of small objects in the dataset it can benefit from training at higher resolutions such as --img 1280. If there are many small objects then custom datasets will benefit from training at native or higher resolution. Best inference results are obtained at the same --img as the training was run at, i.e. if you train at --img 1280 you should also test and detect at --img 1280.
Batch size. Use the largest --batch-size that your hardware allows for. Small batch sizes produce poor batchnorm statistics and should be avoided.
Hyperparameters. Default hyperparameters are in hyp.scratch-low.yaml. We recommend you train with default hyperparameters first before thinking of modifying any. In general, increasing augmentation hyperparameters will reduce and delay overfitting, allowing for longer trainings and higher final mAP. Reduction in loss component gain hyperparameters like hyp['obj'] will help reduce overfitting in those specific loss components. For an automated method of optimizing these hyperparameters, see our Hyperparameter Evolution Tutorial.

@glenn-jocher Thank you for responding So the low confidence does not have anything to do with the extreme small size of objects right ? I have trained the model for around 100 epoch should how long should I train beyond this point ? I have not seen any drop in obj loss over this epochs and other losses / metrics are stabilized I am attaching training details here

This run was done using most of the settings at default probably cutmix, copypaste were set to 0.0 and mosaic set to 0.3 rest was left untouched. also, would you recommend training using a pretrained as a starting point or I would get better results using training from scratch ?

labels.jpg labels labels_correlogram.jpg P curve P_curve R curve R_curve pr curve PR_curve f1 curve F1_curve results.png results confusion matrix confusion_matrix

glenn-jocher commented 2 years ago

@PushpakBhoge for very small objects you'll benefit most from a larger train/detect --imgsz, i.e. 1280 or higher.

PushpakBhoge commented 2 years ago

yes that's why I am preserving the resolution of the original image I am not resizing image in any way. I am taking crops of 512x512 from original images using SAHI, each image provides 6-12 crops I basically tiled all the dataset like this and doing same thing at inference as well

glenn-jocher commented 2 years ago

@PushpakBhoge increased image size helps small objects even if expanded beyond native size. COCO performs better at 1280 even though no image is larger than 640.

Alternatively you could try a P2 model for small objects that goes down to stride 4: https://github.com/ultralytics/yolov5/blob/master/models/hub/yolov5-p2.yaml

PushpakBhoge commented 2 years ago

hmmm so basically due to the small size the actual area of the object is small which may not be enough for the object loss to converge, if we increased the resolution beyond native for example 512x512 to 1280x1280 the same object will be enlarged and the network, more pixels will contribute to the decision hence loss will go down is my understanding correct?

also what would be the pretrained model argument for this p2 model in --weights will it be yolov5p2.pt ?

glenn-jocher commented 2 years ago

@PushpakBhoge there's simply more for the convolutions to work with if there are more pixels representing the object. Like I said the P2 models are intended for smaller stride outputs down to stride 4.

PushpakBhoge commented 2 years ago

ok thank you for all the help! one last thing is there coco pretrained model available for p2 model ?

glenn-jocher commented 2 years ago

@PushpakBhoge no, but you can pair any --weights with any --cfg. Any layers with matching names and shapes will transfer.

PushpakBhoge512 commented 2 years ago

ok thanks that will do thank you so much for the help!

Akhp888 commented 2 years ago

@glenn-jocher I see Sahi wrapper ( sliding window) on yolov5 being used by many , wanted to know if that impacts the yolov5 results in any way and if thats recommended .

glenn-jocher commented 2 years ago

@Akhp888 I'd consult the SAHI author

github-actions[bot] commented 2 years ago

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Wiki – https://github.com/ultralytics/yolov5/wiki
Tutorials – https://docs.ultralytics.com/yolov5
Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

Ultralytics HUB – https://ultralytics.com/hub
Vision API – https://ultralytics.com/yolov5
About Us – https://ultralytics.com/about
Join Our Team – https://ultralytics.com/work
Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

github-actions[bot] commented 2 years ago

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Wiki – https://github.com/ultralytics/yolov5/wiki
Tutorials – https://docs.ultralytics.com/yolov5
Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

Ultralytics HUB – https://ultralytics.com/hub
Vision API – https://ultralytics.com/yolov5
About Us – https://ultralytics.com/about
Join Our Team – https://ultralytics.com/work
Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

ExtReMLapin commented 2 years ago

@PushpakBhoge have you managed to improve your metrics, and by how much ?

PushpakBhoge commented 2 years ago

@ExtReMLapin I have tried with higher resolution training didn't really have much impact in confidence score in my case. But longer training did help it increased matrices on higher confidence thresholds by thresholds betn 0.3-0.5 I observed improvement of approx 15% while for thresholds Between 0.5 to 0.7 I observed improvement of 40-60%

glenn-jocher commented 2 years ago

Yes if confidence scores are low this in an indicator of too-short training epochs.

ExtReMLapin commented 2 years ago

Alright, i'll give a try, thank you both of you !

github-actions[bot] commented 1 year ago

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Wiki – https://github.com/ultralytics/yolov5/wiki
Tutorials – https://docs.ultralytics.com/yolov5
Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

Ultralytics HUB – https://ultralytics.com/hub
Vision API – https://ultralytics.com/yolov5
About Us – https://ultralytics.com/about
Join Our Team – https://ultralytics.com/work
Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

CesareDavidePace commented 1 year ago

@ExtReMLapin i have the same problem. did you solve it then?

ExtReMLapin commented 1 year ago

I was annotating minutiaes , on big images, so there was two ways of fixing this, boosting the annotations size by 3 (and dividing by 3 at inference) and doing tiling, which means you don't reduce your big image size, you just subdivise it in 1024x1024 chunks and feed each chunks one by one.

P2 Model didn't really help me there.

CesareDavidePace commented 1 year ago

what does it mean boosting the annotations size by 3?

ExtReMLapin commented 1 year ago

I had all my annotations and I decided to do : width = width*3, same for height.

Show us some sample of your dataset so we can judge and see how to improve it

CesareDavidePace commented 1 year ago

I can't show the dataset, but the problem is that I can't get good confidence in the results. For me, these are large images and I do the tiling 1920x1920. I also share the hyp used.

results F1_curve

lr0: 0.01 lrf: 0.1 momentum: 0.937 weight_decay: 0.0005 warmup_epochs: 3.0 warmup_momentum: 0.8 warmup_bias_lr: 0.1 box: 0.05 cls: 0.3 cls_pw: 1.0 obj: 0.5 obj_pw: 1.0 iou_t: 0.2 anchor_t: 4.0 fl_gamma: 0.0 hsv_h: 0.015 hsv_s: 0.7 hsv_v: 0.4 degrees: 10 translate: 0.1 scale: 1 shear: 3 perspective: 0.0 flipud: 0.0 fliplr: 0.5 mosaic: 1.0 mixup: 0.2 copy_paste: 0.2

I've already tried to train with multiple epochs but the map immediately goes into overfitting. I can't solve this problem even by lowering the obj loss and increasing the data augment

github-actions[bot] commented 1 year ago

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Wiki – https://github.com/ultralytics/yolov5/wiki
Tutorials – https://docs.ultralytics.com/yolov5
Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

Ultralytics HUB – https://ultralytics.com/hub
Vision API – https://ultralytics.com/yolov5
About Us – https://ultralytics.com/about
Join Our Team – https://ultralytics.com/work
Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

RajibChanda commented 1 year ago

@PushpakBhoge what was your batch size during training? Did you trained with batch size of 1 or some other batch size?

ultralytics / yolov5