ultralytics / yolov5

YOLOv5 πŸš€ in PyTorch > ONNX > CoreML > TFLite
https://docs.ultralytics.com
GNU Affero General Public License v3.0
50.55k stars 16.3k forks source link

Worse confidence scores on small Objects detect #8906

Closed PushpakBhoge closed 1 year ago

PushpakBhoge commented 2 years ago

Search before asking

Question

Hi

I am trying to train a model on custom dataset, the dataset has only one class and very small objects. To give you an idea at any time on a image of 1200x1200 highest object size is 64x64 most of the boxes are below 32x32

I have tiled my dataset with patch size of 512x512 using SAHI crop I did this to preserve the resolution which is necessary for small Objects

The dataset is huge around 100k images. after tiling there are 75% images with atleast one object and 25% negative samples. I am also confident about the quality of annotations error rate in annotations is less than 5%

I have trained it with different set of params and long schedules the mAP is good, bounding boxes are also on spot most of the times

Problem I am facing is all the predictions have very low confidence score I am talking about 0.05 hardly any bbox has score higher than 0.25, anything higher than 0.1 is rare

in training as well I evolved params over 60 generations on a small subset of data to get best hyper parameters, Object loss is not decreasing. At epoch 1 bbox loss was 0.13 and object loss was 0.10 after 50 iterations bbox loss went down to 0.068 and mAP0.5 went up to 28. I have used pretrained v6 weights as a starting point, haven't tried training from scratch.q

But the object loss didn't budge, lowest I have seen is 0.998 that's it

What could be going wrong here ?

Additional

No response

github-actions[bot] commented 2 years ago

πŸ‘‹ Hello @PushpakBhoge, thank you for your interest in YOLOv5 πŸš€! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a πŸ› Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Status

CI CPU testing

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

glenn-jocher commented 2 years ago

@PushpakBhoge πŸ‘‹ Hello! Thanks for asking about improving YOLOv5 πŸš€ training results. Low confidence scores are indicative of too short trainings and can be resolved by training for more epochs.

Most of the time good results can be obtained with no changes to the models or training settings, provided your dataset is sufficiently large and well labelled. If at first you don't get good results, there are steps you might be able to take to improve, but we always recommend users first train with all default settings before considering any changes. This helps establish a performance baseline and spot areas for improvement.

If you have questions about your training results we recommend you provide the maximum amount of information possible if you expect a helpful response, including results plots (train losses, val losses, P, R, mAP), PR curve, confusion matrix, training mosaics, test results and dataset statistics images such as labels.png. All of these are located in your project/name directory, typically yolov5/runs/train/exp.

We've put together a full guide for users looking to get the best results on their YOLOv5 trainings below.

Dataset

COCO Analysis

Model Selection

Larger models like YOLOv5x and YOLOv5x6 will produce better results in nearly all cases, but have more parameters, require more CUDA memory to train, and are slower to run. For mobile deployments we recommend YOLOv5s/m, for cloud deployments we recommend YOLOv5l/x. See our README table for a full comparison of all models.

YOLOv5 Models

Training Settings

Before modifying anything, first train with default settings to establish a performance baseline. A full list of train.py settings can be found in the train.py argparser.

Further Reading

If you'd like to know more a good place to start is Karpathy's 'Recipe for Training Neural Networks', which has great ideas for training that apply broadly across all ML domains: http://karpathy.github.io/2019/04/25/recipe/

Good luck πŸ€ and let us know if you have any other questions!

PushpakBhoge commented 2 years ago

@glenn-jocher Thank you for responding So the low confidence does not have anything to do with the extreme small size of objects right ? I have trained the model for around 100 epoch should how long should I train beyond this point ? I have not seen any drop in obj loss over this epochs and other losses / metrics are stabilized I am attaching training details here

This run was done using most of the settings at default probably cutmix, copypaste were set to 0.0 and mosaic set to 0.3 rest was left untouched. also, would you recommend training using a pretrained as a starting point or I would get better results using training from scratch ?

labels.jpg labels labels_correlogram.jpg labels_correlogram P curve P_curve R curve R_curve pr curve PR_curve f1 curve F1_curve results.png results confusion matrix confusion_matrix

glenn-jocher commented 2 years ago

@PushpakBhoge for very small objects you'll benefit most from a larger train/detect --imgsz, i.e. 1280 or higher.

PushpakBhoge commented 2 years ago

yes that's why I am preserving the resolution of the original image I am not resizing image in any way. I am taking crops of 512x512 from original images using SAHI, each image provides 6-12 crops I basically tiled all the dataset like this and doing same thing at inference as well

glenn-jocher commented 2 years ago

@PushpakBhoge increased image size helps small objects even if expanded beyond native size. COCO performs better at 1280 even though no image is larger than 640.

Alternatively you could try a P2 model for small objects that goes down to stride 4: https://github.com/ultralytics/yolov5/blob/master/models/hub/yolov5-p2.yaml

PushpakBhoge commented 2 years ago

hmmm so basically due to the small size the actual area of the object is small which may not be enough for the object loss to converge, if we increased the resolution beyond native for example 512x512 to 1280x1280 the same object will be enlarged and the network, more pixels will contribute to the decision hence loss will go down is my understanding correct?

also what would be the pretrained model argument for this p2 model in --weights will it be yolov5p2.pt ?

glenn-jocher commented 2 years ago

@PushpakBhoge there's simply more for the convolutions to work with if there are more pixels representing the object. Like I said the P2 models are intended for smaller stride outputs down to stride 4.

PushpakBhoge commented 2 years ago

ok thank you for all the help! one last thing is there coco pretrained model available for p2 model ?

glenn-jocher commented 2 years ago

@PushpakBhoge no, but you can pair any --weights with any --cfg. Any layers with matching names and shapes will transfer.

PushpakBhoge512 commented 2 years ago

ok thanks that will do thank you so much for the help!

Akhp888 commented 2 years ago

@glenn-jocher I see Sahi wrapper ( sliding window) on yolov5 being used by many , wanted to know if that impacts the yolov5 results in any way and if thats recommended .

glenn-jocher commented 2 years ago

@Akhp888 I'd consult the SAHI author

github-actions[bot] commented 2 years ago

πŸ‘‹ Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 πŸš€ resources:

Access additional Ultralytics ⚑ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐!

github-actions[bot] commented 2 years ago

πŸ‘‹ Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 πŸš€ resources:

Access additional Ultralytics ⚑ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐!

ExtReMLapin commented 2 years ago

@PushpakBhoge have you managed to improve your metrics, and by how much ?

PushpakBhoge commented 2 years ago

@ExtReMLapin I have tried with higher resolution training didn't really have much impact in confidence score in my case. But longer training did help it increased matrices on higher confidence thresholds by thresholds betn 0.3-0.5 I observed improvement of approx 15% while for thresholds Between 0.5 to 0.7 I observed improvement of 40-60%

glenn-jocher commented 2 years ago

Yes if confidence scores are low this in an indicator of too-short training epochs.

ExtReMLapin commented 2 years ago

Alright, i'll give a try, thank you both of you !

github-actions[bot] commented 1 year ago

πŸ‘‹ Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 πŸš€ resources:

Access additional Ultralytics ⚑ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐!

CesareDavidePace commented 1 year ago

@ExtReMLapin i have the same problem. did you solve it then?

ExtReMLapin commented 1 year ago

I was annotating minutiaes , on big images, so there was two ways of fixing this, boosting the annotations size by 3 (and dividing by 3 at inference) and doing tiling, which means you don't reduce your big image size, you just subdivise it in 1024x1024 chunks and feed each chunks one by one.

P2 Model didn't really help me there.

CesareDavidePace commented 1 year ago

what does it mean boosting the annotations size by 3?

ExtReMLapin commented 1 year ago

I had all my annotations and I decided to do : width = width*3, same for height.

Show us some sample of your dataset so we can judge and see how to improve it

CesareDavidePace commented 1 year ago

I can't show the dataset, but the problem is that I can't get good confidence in the results. For me, these are large images and I do the tiling 1920x1920. I also share the hyp used.

results F1_curve

lr0: 0.01 lrf: 0.1 momentum: 0.937 weight_decay: 0.0005 warmup_epochs: 3.0 warmup_momentum: 0.8 warmup_bias_lr: 0.1 box: 0.05 cls: 0.3 cls_pw: 1.0 obj: 0.5 obj_pw: 1.0 iou_t: 0.2 anchor_t: 4.0 fl_gamma: 0.0 hsv_h: 0.015 hsv_s: 0.7 hsv_v: 0.4 degrees: 10 translate: 0.1 scale: 1 shear: 3 perspective: 0.0 flipud: 0.0 fliplr: 0.5 mosaic: 1.0 mixup: 0.2 copy_paste: 0.2

I've already tried to train with multiple epochs but the map immediately goes into overfitting. I can't solve this problem even by lowering the obj loss and increasing the data augment

github-actions[bot] commented 1 year ago

πŸ‘‹ Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 πŸš€ resources:

Access additional Ultralytics ⚑ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐!

RajibChanda commented 1 year ago

@PushpakBhoge what was your batch size during training? Did you trained with batch size of 1 or some other batch size?