Different results from detect vs val, "no detections"

[PeterKBailey]

Discussed in https://github.com/ultralytics/yolov5/discussions/12861

I'm sorry to raise this as an issue but I'm not sure what else to try, please let me know if this belongs somewhere else!! I just clicked on the option given by GitHub.

[I am using this dataset: https://www.mapillary.com/dataset/trafficsign]

^{Originally posted by **PeterKBailey** March 29, 2024} Hello! I hope I'm asking this in the right place but I'm new to YOLO and have been trying to determine if I have properly built and structured my dataset. I used the following command to first train the model on a very small data subset (222 images) `python yolov5/train.py --img 640 --batch 16 --epochs 3 --data dataset.yaml --weights '' --cfg yolov5n.yaml` I also included a validation directory which contains the exact same images in my training set. All I'm trying to do right now is see that I can make a model which overfits / perfectly predicts for this training data. After having completed the training I did two things: `python detect.py --weights best.pt --img 640 --data dataset.yaml --source image.jpg --conf-thres 0.01` and `python .val.py --weights best.pt --img 640 --data dataset.yaml` with the following results: (output from detect on left, and output from val on the right)  Please excuse the resolution, this is a cropped in screenshot but is only meant to show that the validation batch worked as expected but that detection on the image apparently did not. Things I have tried: - I have tried lowering and lowering the conf-thres but nothing happens until suddenly the entire image is taken up by spurious bounding box predictions. - I did a training run of 15 epochs, everything else the same, no difference in validation vs detection - I did a training run using my dataset but beginning with the preset YOLOv5n weights, still no difference I'm not sure where to go from here so advice would be greatly appreciated! Thanks! OS: Windows 10 YOLOv5 v7.0-295-gac6c4383 Python-3.11.6 torch-2.2.1+cpu CPU Edit with additional things I have tried: - Replacing the input image to detect with a resized 640x640 image (both by squashing/stretching and by letterboxing) - Training a yolov5s model instead (val still works, detect does not) - setting conf-thres to 0.001 and iou to 0.6 when detecting (which are apparently the defaults for validation), no detections. - Upped dataset size to 4130 training images with 950 validation images, still validation shows results but detection does not I have also double checked my bounding boxes with [the YoloBBoxChecker](https://github.com/ivder/YoloBBoxChecker) and am certain that my text files are correct. I didn't doubt this particularly given that val is successful but figured it was one more item on the list. 

[github-actions[bot]]

👋 Hello @PeterKBailey, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@PeterKBailey hello PeterKBailey,

Thank you for reaching out and providing a detailed account of the issue you're encountering. It's great that you're taking the time to experiment with the YOLOv5 models and datasets.

From your description, it seems like the model is training well but you're experiencing a discrepancy between validation and detection results. This issue is typically related to how the model generalizes to new data or how detection settings are applied.

A few things to consider:

• Ensure your --source in detect.py is representative of the data the model was trained on. A dramatic shift in data distribution between training and detection can lead to poor performance.
• Since you've tried adjusting the conf-thres and iou-thres without success, the issue might not lie with the confidence or IOU thresholds. However, it's good to note that detecting with very low conf-thres can indeed produce a lot of spurious predictions.
• Given validation works but detection does not, double-check the preprocessing steps in detection. The model expects images to be processed in the same way during both training/validation and detection. This includes resizing without distorting aspect ratio (typically done by padding to maintain the aspect ratio).

Considering you've already tried training with more images and different model sizes, the issue seems to focus more on the detection stage rather than training. I would recommend revisiting the preprocessing steps in detect.py to ensure they align with how your validation set images are processed.

Unfortunately, without seeing the exact images or data you're working with in detect.py, it's challenging to provide a more precise diagnosis. I encourage you to review the preprocessing part of the detection pipeline and ensure it closely matches that of your training and validation pipeline.

If you continue to face difficulties, please provide more details about the step-by-step preprocessing and detection commands used, along with any specific error messages or unexpected output details. This additional information would help in pinpointing the exact cause of the discrepancy.

Keep experimenting and asking questions; that's how we all learn and improve. Good luck! 🚀

[PeterKBailey]

@glenn-jocher Hi glenn-jocher,

Thanks for your reply! I have a few follow up questions and I'll also redo the training and document what I'm doing while I go:

So first to try and respond to your suggestions:

• Ensure your --source in detect.py is representative of the data

  • I am using one of my training images as the image being detected on, I wanted to see that the model could overfit my dataset

• Given validation works but detection does not, double-check the preprocessing steps in detection.

  • Sorry, I'm not sure if you mean I should look into how the source code operates? Or do I need to do my own preprocessing when I'm using detect that I don't need to do with val?

1) Training:

So when it comes to preprocessing, I'm not doing anything myself, nor am I using Roboflow or any other pipeline. My dataset is a set of jpg images with varying resolutions (ex: 5312x2988, 4032x3024, 4160x3120, 1920x1080). I have 114 such images in my images/training directory. I have 74 different images in my images/validation directory.

One example training image /images/training/0VE6VzyUjItYMGIsRKwJBg.jpg:

The corresponding training label /labels/training/0VE6VzyUjItYMGIsRKwJBg.txt:

0 0.6611328125 0.5291748046875 0.01123046875 0.019775390625
89 0.6590576171875 0.486328125 0.030029296875 0.0517578125

Image + BBoxes using YoloBBoxChecker

To verify that my .txt files are correct I visualize on an example:

Now I use the following command:

python yolov5/train.py --img 640 --batch 16 --epochs 3 --data dataset.yaml --weights '' --cfg yolov5n.yaml

I get the following output:

train: weights=, cfg=yolov5n.yaml, data=dataset.yaml, hyp=yolov5\data\hyps\hyp.scratch-low.yaml, epochs=3, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, noplots=False, evolve=None, evolve_population=yolov5\data\hyps, resume_evolve=None, bucket=, cache=None, image_weights=False, device=, multi_scale=False, single_cls=False, optimizer=SGD, sync_bn=False, workers=8, project=yolov5\runs\train, name=exp, exist_ok=False, quad=False, cos_lr=False, label_smoothing=0.0, patience=100, freeze=[0], save_period=-1, seed=0, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest, ndjson_console=False, ndjson_file=False
github: up to date with https://github.com/ultralytics/yolov5
YOLOv5  v7.0-295-gac6c4383 Python-3.11.6 torch-2.2.1+cpu CPU

hyperparameters: lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0

With these results:

My dataset.yaml looks like:

Expand

```yaml names: - other-sign - regulatory--keep-right--g1 - regulatory--priority-over-oncoming-vehicles--g1 - regulatory--height-limit--g1 - regulatory--maximum-speed-limit-35--g2 - warning--railroad-crossing-with-barriers--g1 - warning--curve-left--g2 - warning--falling-rocks-or-debris-right--g1 - regulatory--keep-right--g4 - warning--pedestrians-crossing--g4 - complementary--go-right--g2 - complementary--keep-left--g1 - regulatory--maximum-speed-limit-45--g3 - complementary--chevron-right--g3 - regulatory--one-way-right--g2 - regulatory--yield--g1 - regulatory--one-way-straight--g1 - warning--curve-right--g1 - regulatory--pedestrians-only--g2 - information--emergency-facility--g2 - regulatory--no-entry--g1 - warning--railroad-crossing--g3 - warning--pedestrians-crossing--g5 - warning--crossroads--g3 - complementary--chevron-left--g5 - information--motorway--g1 - regulatory--no-stopping--g15 - information--pedestrians-crossing--g1 - warning--railroad-crossing-without-barriers--g3 - regulatory--go-straight-or-turn-right--g1 - complementary--go-right--g1 - complementary--distance--g1 - warning--slippery-road-surface--g1 - warning--curve-left--g1 - information--parking--g1 - complementary--go-left--g1 - information--tram-bus-stop--g2 - warning--crossroads--g1 - regulatory--no-overtaking--g2 - warning--railroad-crossing-with-barriers--g2 - complementary--one-direction-left--g1 - regulatory--stop--g1 - complementary--trucks-turn-right--g1 - regulatory--maximum-speed-limit-30--g1 - regulatory--priority-road--g4 - regulatory--pedestrians-only--g1 - warning--pedestrians-crossing--g9 - warning--junction-with-a-side-road-acute-right--g1 - regulatory--end-of-maximum-speed-limit-30--g2 - information--end-of-living-street--g1 - regulatory--one-way-right--g3 - information--road-bump--g1 - warning--height-restriction--g2 - complementary--obstacle-delineator--g2 - warning--double-curve-first-left--g2 - regulatory--no-overtaking--g5 - information--food--g2 - warning--divided-highway-ends--g2 - regulatory--turn-right--g1 - complementary--chevron-left--g1 - regulatory--turn-left--g1 - regulatory--no-parking-or-no-stopping--g3 - warning--roundabout--g1 - regulatory--no-heavy-goods-vehicles--g1 - regulatory--maximum-speed-limit-60--g1 - complementary--maximum-speed-limit-70--g1 - regulatory--maximum-speed-limit-40--g1 - warning--road-widens--g1 - complementary--chevron-right--g1 - warning--road-bump--g1 - warning--uneven-road--g6 - regulatory--maximum-speed-limit-50--g1 - regulatory--no-parking--g5 - regulatory--turn-left--g3 - warning--railroad-crossing-without-barriers--g1 - warning--junction-with-a-side-road-perpendicular-right--g3 - regulatory--maximum-speed-limit-100--g1 - warning--double-curve-first-right--g1 - regulatory--maximum-speed-limit-5--g1 - complementary--extent-of-prohibition-area-both-direction--g1 - warning--road-narrows-left--g2 - warning--children--g2 - information--parking--g5 - regulatory--no-u-turn--g3 - warning--y-roads--g1 - warning--trail-crossing--g2 - regulatory--maximum-speed-limit-40--g3 - regulatory--go-straight-or-turn-left--g1 - regulatory--bicycles-only--g1 - warning--texts--g2 - regulatory--one-way-left--g1 - warning--road-narrows-right--g2 - regulatory--one-way-left--g3 - regulatory--give-way-to-oncoming-traffic--g1 - warning--double-curve-first-right--g2 - complementary--maximum-speed-limit-30--g1 - regulatory--no-u-turn--g1 - warning--narrow-bridge--g1 - regulatory--turn-right-ahead--g1 - information--parking--g3 - regulatory--maximum-speed-limit-70--g1 - warning--uneven-road--g2 - regulatory--shared-path-pedestrians-and-bicycles--g1 - regulatory--pass-on-either-side--g2 - regulatory--no-bicycles--g2 - regulatory--no-pedestrians--g2 - regulatory--no-stopping--g2 - complementary--maximum-speed-limit-15--g1 - warning--roundabout--g25 - regulatory--go-straight-or-turn-left--g2 - regulatory--no-parking--g2 - regulatory--u-turn--g1 - regulatory--keep-left--g1 - regulatory--go-straight--g1 - regulatory--keep-right--g2 - regulatory--no-overtaking--g1 - regulatory--no-parking-or-no-stopping--g2 - information--telephone--g2 - regulatory--road-closed-to-vehicles--g3 - regulatory--no-left-turn--g3 - warning--other-danger--g3 - information--airport--g1 - regulatory--no-right-turn--g1 - regulatory--no-left-turn--g1 - warning--railroad-crossing-without-barriers--g4 - warning--texts--g1 - information--end-of-built-up-area--g1 - warning--junction-with-a-side-road-acute-left--g1 - warning--divided-highway-ends--g1 - regulatory--maximum-speed-limit-90--g1 - regulatory--maximum-speed-limit-110--g1 - warning--junction-with-a-side-road-perpendicular-left--g4 - warning--other-danger--g1 - regulatory--no-parking--g1 - warning--hairpin-curve-left--g3 - information--bus-stop--g1 - warning--winding-road-first-left--g1 - warning--turn-right--g1 - regulatory--no-bicycles--g1 - regulatory--no-heavy-goods-vehicles--g4 - regulatory--weight-limit--g1 - regulatory--radar-enforced--g1 - regulatory--lane-control--g1 - regulatory--turn-right--g2 - warning--traffic-signals--g3 - warning--added-lane-right--g1 - warning--emergency-vehicles--g1 - complementary--keep-right--g1 - complementary--distance--g3 - warning--winding-road-first-right--g3 - warning--traffic-signals--g1 - complementary--both-directions--g1 - warning--junction-with-a-side-road-perpendicular-right--g1 - regulatory--stop--g10 - regulatory--maximum-speed-limit-20--g1 - regulatory--maximum-speed-limit-25--g2 - regulatory--no-motor-vehicles-except-motorcycles--g2 - complementary--maximum-speed-limit-25--g1 - complementary--maximum-speed-limit-55--g1 - warning--curve-right--g2 - regulatory--no-pedestrians--g1 - complementary--maximum-speed-limit-35--g1 - complementary--chevron-left--g3 - regulatory--wrong-way--g1 - complementary--chevron-left--g2 - warning--double-reverse-curve-right--g1 - warning--double-curve-first-left--g1 - regulatory--maximum-speed-limit-30--g3 - regulatory--no-bicycles--g3 - regulatory--no-heavy-goods-vehicles--g2 - warning--traffic-merges-right--g1 - information--limited-access-road--g1 - regulatory--maximum-speed-limit-55--g2 - complementary--maximum-speed-limit-45--g1 - warning--junction-with-a-side-road-perpendicular-left--g3 - warning--pass-left-or-right--g2 - complementary--one-direction-right--g1 - regulatory--turn-left--g2 - regulatory--stop--g2 - information--pedestrians-crossing--g2 - regulatory--maximum-speed-limit-80--g1 - complementary--trucks--g1 - complementary--tow-away-zone--g1 - warning--roadworks--g1 - regulatory--turn-left-ahead--g1 - warning--horizontal-alignment-right--g1 - warning--trams-crossing--g1 - warning--double-turn-first-right--g1 - warning--narrow-bridge--g3 - warning--children--g1 - warning--domestic-animals--g3 - warning--winding-road-first-right--g1 - information--central-lane--g1 - regulatory--road-closed--g2 - regulatory--no-vehicles-carrying-dangerous-goods--g1 - warning--t-roads--g2 - information--minimum-speed-40--g1 - warning--school-zone--g2 - regulatory--reversible-lanes--g2 - regulatory--no-parking-or-no-stopping--g1 - warning--traffic-merges-right--g2 - complementary--maximum-speed-limit-20--g1 - warning--slippery-road-surface--g2 - warning--traffic-signals--g2 - regulatory--one-way-left--g2 - warning--bus-stop-ahead--g3 - regulatory--no-u-turn--g2 - regulatory--no-overtaking--g4 - regulatory--keep-left--g2 - information--stairs--g1 - warning--two-way-traffic--g1 - regulatory--no-turn-on-red--g1 - warning--turn-right--g2 - warning--road-narrows-right--g1 - complementary--turn-left--g2 - warning--texts--g3 - information--end-of-motorway--g1 - regulatory--pass-on-either-side--g1 - complementary--chevron-right--g4 - regulatory--no-left-turn--g2 - complementary--chevron-right--g5 - warning--trucks-crossing--g1 - regulatory--no-motor-vehicle-trailers--g1 - warning--road-bump--g2 - regulatory--no-stopping--g8 - regulatory--maximum-speed-limit-led-100--g1 - complementary--obstacle-delineator--g1 - regulatory--maximum-speed-limit-10--g1 - complementary--priority-route-at-intersection--g1 - regulatory--maximum-speed-limit-40--g6 - regulatory--maximum-speed-limit-45--g1 - regulatory--one-way-right--g1 - regulatory--end-of-bicycles-only--g1 - regulatory--roundabout--g1 - information--living-street--g1 - complementary--except-bicycles--g1 - warning--bicycles-crossing--g1 - warning--pedestrian-stumble-train--g1 - regulatory--no-turn-on-red--g2 - complementary--maximum-speed-limit-75--g1 - information--safety-area--g2 - warning--turn-left--g1 - regulatory--road-closed--g1 - warning--stop-ahead--g9 - regulatory--mopeds-and-bicycles-only--g1 - regulatory--end-of-speed-limit-zone--g1 - information--interstate-route--g1 - complementary--distance--g2 - warning--roadworks--g3 - complementary--chevron-left--g4 - regulatory--triple-lanes-turn-left-center-lane--g1 - warning--roadworks--g4 - information--highway-exit--g1 - regulatory--turn-right--g3 - warning--winding-road-first-left--g2 - warning--flaggers-in-road--g1 - regulatory--no-motor-vehicles--g1 - regulatory--no-right-turn--g2 - regulatory--left-turn-yield-on-green--g1 - regulatory--dual-lanes-go-straight-on-right--g1 - regulatory--no-overtaking-by-heavy-goods-vehicles--g1 - warning--pedestrians-crossing--g1 - regulatory--no-straight-through--g1 - complementary--chevron-right-unsure--g6 - warning--offset-roads--g3 - regulatory--maximum-speed-limit-120--g1 - regulatory--go-straight-or-turn-right--g3 - information--disabled-persons--g1 - information--parking--g6 - warning--loop-270-degree--g1 - regulatory--dual-path-bicycles-and-pedestrians--g1 - regulatory--buses-only--g1 - complementary--accident-area--g3 - complementary--pass-right--g1 - warning--dual-lanes-right-turn-or-go-straight--g1 - warning--road-narrows--g1 - information--children--g1 - regulatory--end-of-prohibition--g1 - information--bike-route--g1 - information--end-of-limited-access-road--g1 - regulatory--no-mopeds-or-bicycles--g1 - warning--wombat-crossing--g1 - warning--crossroads-with-priority-to-the-right--g1 - regulatory--maximum-speed-limit-led-80--g1 - information--highway-interstate-route--g2 - regulatory--stop-here-on-red-or-flashing-light--g1 - warning--traffic-merges-left--g1 - warning--hairpin-curve-right--g1 - warning--equestrians-crossing--g2 - information--gas-station--g3 - regulatory--keep-right--g6 - warning--road-widens-right--g1 - warning--wild-animals--g4 - regulatory--turn-right-ahead--g2 - information--trailer-camping--g1 - warning--railroad-crossing--g1 - warning--domestic-animals--g1 - warning--playground--g1 - regulatory--no-stopping--g5 - regulatory--end-of-maximum-speed-limit-70--g2 - warning--traffic-merges-left--g2 - regulatory--no-motorcycles--g1 - information--hospital--g1 - regulatory--no-stopping--g4 - warning--falling-rocks-or-debris-right--g4 - regulatory--shared-path-bicycles-and-pedestrians--g1 - warning--railroad-intersection--g3 - regulatory--minimum-safe-distance--g1 - warning--steep-ascent--g7 - warning--kangaloo-crossing--g1 - warning--hairpin-curve-left--g1 - regulatory--go-straight--g3 - information--dead-end--g1 - complementary--turn-right--g2 - regulatory--stop-signals--g1 - warning--falling-rocks-or-debris-right--g2 - regulatory--passing-lane-ahead--g1 - information--airport--g2 - regulatory--no-turn-on-red--g3 - warning--junction-with-a-side-road-perpendicular-left--g1 - regulatory--width-limit--g1 - information--gas-station--g1 - regulatory--go-straight-or-turn-left--g3 - information--camp--g1 - regulatory--no-motorcycles--g2 - regulatory--stop-here-on-red-or-flashing-light--g2 - regulatory--no-turns--g1 - regulatory--maximum-speed-limit-15--g1 - regulatory--no-straight-through--g2 - regulatory--maximum-speed-limit-led-60--g1 - regulatory--maximum-speed-limit-100--g3 - warning--wild-animals--g1 - regulatory--no-motor-vehicles-except-motorcycles--g1 - complementary--buses--g1 - regulatory--parking-restrictions--g2 - regulatory--bicycles-only--g3 - regulatory--end-of-buses-only--g1 - warning--two-way-traffic--g2 - regulatory--end-of-priority-road--g1 - information--no-parking--g3 - information--telephone--g1 - regulatory--truck-speed-limit-60--g1 - warning--horizontal-alignment-left--g1 - warning--railroad-crossing--g4 - information--parking--g2 - warning--slippery-motorcycles--g1 - regulatory--maximum-speed-limit-50--g6 - warning--pedestrians-crossing--g12 - regulatory--do-not-block-intersection--g1 - regulatory--end-of-maximum-speed-limit-70--g1 - complementary--maximum-speed-limit-40--g1 - regulatory--dual-lanes-go-straight-on-left--g1 - warning--horizontal-alignment-right--g3 - regulatory--end-of-no-parking--g1 - warning--pedestrians-crossing--g10 - warning--t-roads--g1 - regulatory--detour-left--g1 - warning--road-narrows-left--g1 - warning--bicycles-crossing--g2 - regulatory--dual-lanes-turn-left-or-straight--g1 - regulatory--do-not-stop-on-tracks--g1 - warning--roadworks--g2 - warning--dip--g2 - regulatory--maximum-speed-limit-65--g2 - warning--road-narrows--g2 - regulatory--no-heavy-goods-vehicles--g5 - regulatory--road-closed-to-vehicles--g1 - warning--railroad-intersection--g4 - warning--railroad-crossing-with-barriers--g4 - regulatory--no-pedestrians--g3 - regulatory--maximum-speed-limit-25--g1 - regulatory--text-four-lines--g1 - regulatory--no-buses--g3 - regulatory--bicycles-only--g2 - warning--bicycles-crossing--g3 - warning--uneven-roads-ahead--g1 - warning--traffic-signals--g4 - regulatory--no-pedestrians-or-bicycles--g1 - information--lodging--g1 - warning--shared-lane-motorcycles-bicycles--g1 - regulatory--dual-lanes-turn-left-no-u-turn--g1 - regulatory--no-hawkers--g1 - regulatory--roundabout--g2 - regulatory--weight-limit-with-trucks--g1 - information--parking--g45 - regulatory--dual-path-pedestrians-and-bicycles--g1 - regulatory--no-heavy-goods-vehicles-or-buses--g1 - regulatory--no-motor-vehicles--g4 - warning--pedestrians-crossing--g11 - warning--hairpin-curve-right--g4 - warning--accidental-area-unsure--g2 - warning--pass-left-or-right--g1 - warning--restricted-zone--g1 - regulatory--turning-vehicles-yield-to-pedestrians--g1 - information--end-of-pedestrians-only--g2 - regulatory--no-right-turn--g3 - regulatory--dual-lanes-turn-right-or-straight--g1 - complementary--maximum-speed-limit-50--g1 - warning--playground--g3 - warning--roadworks--g6 - information--dead-end-except-bicycles--g1 nc: 401 train: C:\Users\Peter\Downloads\out\data\images\training val: C:\Users\Peter\Downloads\out\data\images\validation ```

2) Validation:

What I want to accomplish here is see that the model works on images that were trained on at the very least.

First I (temporarily) remove all images from labels/validation and from images/validation and place only the single image and label for 0VE6VzyUjItYMGIsRKwJBg

I don't have a command to share, I did this in my file explorer :)

I then ran validation on that image shown above (0VE6VzyUjItYMGIsRKwJBg) with the following command:

python ./yolov5/val.py --weights ./yolov5/runs/train/exp/weights/best.pt --img 640 --data ./dataset.yaml

I get the following image in my runs/val folder:

So everything is looking good up until now!! 😄

3) Detection

Finally I feed that same image to detection (keeping in mind that this image was used in training, I expect this should have the same or similar results to val)

I use the following command:

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp/weights/best.pt --img 640 --data dataset.yaml --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg

I get the following output and image in my runs/detect/exp folder:

detect: weights=['./yolov5/runs/train/exp/weights/best.pt'], source=./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg, data=dataset.yaml, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_csv=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=yolov5\runs\detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False, vid_stride=1
YOLOv5  v7.0-295-gac6c4383 Python-3.11.6 torch-2.2.1+cpu CPU

Fusing layers...
YOLOv5n summary: 157 layers, 2301718 parameters, 0 gradients, 5.8 GFLOPs
image 1/1 C:\Users\Peter\Downloads\out\data\images\training\0VE6VzyUjItYMGIsRKwJBg.jpg: 384x640 (no detections), 149.4ms
Speed: 1.0ms pre-process, 149.4ms inference, 1.0ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5\runs\detect\exp

So we can see no detections as stated in the output.

Trying the same thing with the validation conf and iou thresholds:

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp/weights/best.pt --img 640 --data dataset.yaml --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg --conf-thres 0.001 --iou-thres 0.6 --max-det 300

I see the same results

no detections (no point in reattaching the box-less image)

Trying a conf-thres of 0.00001 I can get the following image:

detect: weights=['./yolov5/runs/train/exp/weights/best.pt'], source=./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg, data=dataset.yaml, imgsz=[640, 640], conf_thres=1e-05, iou_thres=0.6, max_det=300, device=, view_img=False, save_txt=False, save_csv=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=yolov5\runs\detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False, vid_stride=1
YOLOv5  v7.0-295-gac6c4383 Python-3.11.6 torch-2.2.1+cpu CPU

Fusing layers...
YOLOv5n summary: 157 layers, 2301718 parameters, 0 gradients, 5.8 GFLOPs
image 1/1 C:\Users\Peter\Downloads\out\data\images\training\0VE6VzyUjItYMGIsRKwJBg.jpg: 384x640 60 regulatory--no-stopping--g15s, 240 regulatory--bicycles-only--g3s, 156.5ms
Speed: 1.0ms pre-process, 156.5ms inference, 18.2ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5\runs\detect\exp3

Final Thoughts and Thanks

So this is a very long post, I tried to be precise and give the relevant information. If there is more that I can give please let me know I would really like to get this solved.

Thank you for your help and thank you in advance if you can help any further!!

[glenn-jocher]

Hello @PeterKBailey,

Thank you for the detailed follow-up and the efforts you've made to troubleshoot this issue. It appears you've done a thorough job testing various configurations, which is very helpful. Your descriptions and steps are clear, offering a good insight into the problem. Let's address your concerns:

1. Regarding preprocessing in detect.py vs. val.py: The preprocessing should be handled automatically by both scripts. You shouldn't need to manually adjust preprocessing steps unless you're troubleshooting or experimenting with different preprocessing techniques. The fact that validation works but detection does not, with the settings you've described, suggests the issue is not related to the preprocessing.

2. Using Training Images for Detection: Using training images for detection to test overfitting is a valid approach. However, seeing no detections despite this can sometimes be tied to how images are processed or the confidence thresholds (which you've already adjusted).

3. Next Steps: Given that you've tried various conf-thres levels, and you only get predictions at extremely low confidence levels, it might indicate the model is uncertain about its predictions. This can happen if the model hasn't learned the features well enough, despite seeing similar images during training. Since you're using a very low number of epochs (3 in your initial command), this might not be sufficient for the model to learn effectively, even from a small dataset.

Considerations:

• Increase Epochs: Try increasing the number of epochs significantly. Though overfitting for testing purposes can work with a relatively low number, it might require more in this context.
• Review Labels: Double-check that the labels in your training images closely represent the objects you want to detect, although it seems you've already verified this with the YoloBBoxChecker.
• Experiment with Different Models: Although you said no difference was observed by switching between YOLOv5n and YOLOv5s models, training time and capacity differences could impact learning. Using a larger variant (e.g., yolov5m or yolov5l) might improve feature extraction, albeit at the cost of increased computational resources.

I noticed you've done great work ensuring the dataset integrity and experimenting with various configurations. Continuing to adjust the training length and possibly experimenting with the learning rate may provide further insights. These steps are iterative and experimental in nature. Your dedication to resolving this is commendable!

Keep up the great work, and don't hesitate to reach out if you have more questions or updates based on these suggestions. 🚀

[PeterKBailey]

Hello @glenn-jocher,

Thank you for your continuing advice and support! So following what you said I tried two new things:

1. I trained a yolov5n model over 64 epochs overnight

It was on the same small training dataset I mentioned before.

I used this command

python yolov5/train.py --img 640 --batch 16 --epochs 64 --data dataset.yaml --weights '' --cfg yolov5n.yaml

I got the following results image:

Validating this model:

python ./yolov5/val.py --weights ./yolov5/runs/train/exp2/weights/best.pt --img 640 --data ./dataset.yaml I get the following image: Which looks good like the other models I've shown before.

I attempted to run detect:

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp2/weights/best.pt --img 640 --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg --data ./dataset.yaml --conf-thres 0.1 --iou-thres 0.6 --max-det 300

Got (no detections).

So I used lower and lower conf-thresholds until I eventually got these spurious detections:

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp2/weights/best.pt --img 640 --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg --data ./dataset.yaml --conf-thres 0.001 --iou-thres 0.6 --max-det 300

2. I then trained on a large model for 3 epochs this morning:

Still using that same small training set.

I ran the following to do so:

python yolov5/train.py --img 640 --batch 16 --epochs 3 --data dataset.yaml --weights '' --cfg yolov5l.yaml

Which gave me the following results image:

Validating this model:

python ./yolov5/val.py --weights ./yolov5/runs/train/exp3/weights/best.pt --img 640 --data ./dataset.yaml

Which produced: (which looks good too).

And I did a detection on with this new model:

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp3/weights/best.pt --img 640 --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg --data ./dataset.yaml --conf-thres 0.1 --iou-thres 0.6 --max-det 300 (no detections)

Again I lowered the conf thres until I got results with

python ./yolov5/detect.py --weights ./yolov5/runs/train/exp3/weights/best.pt --img 640 --source ./images/training/0VE6VzyUjItYMGIsRKwJBg.jpg --data ./dataset.yaml --conf-thres 0.00001 --iou-thres 0.6 --max-det 300

Which produced more spurious boxes unfortunately.

Concluding remarks

I did my best to follow your suggestions but so far I am still not getting a result. I am very confused as to why this is the case, what is validation doing so differently from detect? Are my images too big / do I need to use a different --img parameter?

I can see that these two trained models are clearly different based only on detect and how/what boxes they place at low enough confidence. But they both perform exactly correctly for validation and I'm left puzzled as ever haha! I suppose I can try training the yolov5l model for more/64 epochs but given the costly nature I am hesitant to do that before consulting you for other options.

Do you have any more suggestions I can try? Thank you again for your time and efforts!!

[glenn-jocher]

Hello again @PeterKBailey,

It's great to see your persistence and thorough experimentation with both the small and large models over different epoch lengths. Your efforts are truly commendable. 🌟 It's indeed puzzling that despite the models producing good validation results, the detection phase still struggles to produce meaningful detections without resorting to extremely low confidence thresholds.

Given the scenario you've described, a few thoughts come to mind:

1. Model's Generalization Ability: Even though validation results look good, it's possible that the model's ability to generalize to unseen data or slightly different conditions (like lighting, angle, etc., even in training images) isn't robust enough. This could be a factor if the dataset is small or not diverse enough.

2. Dataset Diversity and Size: If you haven't already, consider expanding your dataset with more varied examples. Sometimes, more data or artificially augmenting existing data (e.g., using different lighting conditions, rotations) can help the model generalize better.

3. Post-Training Analysis: It can be beneficial to analyze which images or types of images your model struggles with. This can provide insights into potential biases or weaknesses in the dataset. For instance, a confusion matrix can help understand if some classes are being consistently misclassified.

4. Hyperparameters Tuning: Besides the number of epochs and the model size, other hyperparameters might influence the detection phase. You could experiment with different learning rates or even the augmentation strategies during training.

5. Detection Script Check: Ensure the detect.py script processes images in the same manner as the validation phase. Although it should be consistent, it's worth verifying that the same transformations and preprocessing steps are applied.

6. Image Size for Detection: You mentioned the size of your images and questioned the use of the --img parameter. It's worth experimenting with this parameter further. Sometimes, using a larger --img size for detection (consistent with the original image resolution) might help, especially if your dataset consists of high-resolution images.

It's not uncommon to face challenges like these, especially when working on fine-tuning models for specific datasets. Before trying a long and potentially costly training with YOLOv5l for 64 epochs, my advice would be to explore the dataset's diversity and size, along with the model's generalization capabilities and hyperparameters tweaking. Also, consider checking the detect.py for any discrepancies in image processing compared to the training/validation phases.

Your dedication to solving this is impressive, and each step brings more valuable insights. Keep exploring, and feel free to share any further observations or results. 🚀

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐