Custom dataset for multi-label detection in a single bounding box

[yoni-f]

Hello,

I have a dataset that involves images with bounding boxes which have more than 1 label.

For example, I have 5 classes (0-4), then an image might have a bounding box with label "2" and another might have a bounding box with two labels "2, 4" ie it belongs to both categories.

Two questions:

1. Is it even possible for the model to output a single box with multi labels? Or are there changes that need to be made to the code?

2. How would you go about building the labels .txt files?

For a single-label box, the .txt should something like this: 2 0.3 0.3 0.5 0.5

For a multi-label box, how would you go about? 2 4 0.3 0.3 0.5 0.5

Or

2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5

Thanks!

[github-actions[bot]]

Hello @yoni-f, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@yoni-f multi-class output is on by default. This will output the same bounding box multiple times for any class above threshold. Use your second option for the labels: 2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[wzj5133329]

@yoni-f multi-class output is on by default. This will output the same bounding box multiple times for any class above threshold. Use your second option for the labels: 2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5

@glenn-jocher hi,if use multi-labels in this project,do i need to modify the loss function? such as one label.txt: 2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5 in compute_loss() BCEcls(ps[:, 5:], t) t: [0,0,1,0,0] but really it is [0,0,1,0,1] in BCEWithLogitsLoss?

[glenn-jocher]

@wzj5133329 no loss function modifications are required.

[wzj5133329]

@wzj5133329 no loss function modifications are required.

@glenn-jocher ok, i will have a try,thank you!

[KrystalCWT]

Hi, I tried to modify the label.txt to '2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5'

but when I detect, each detected box is only classified to one class. Do you know how to fix it?

[wzj5133329]

Hi, I tried to modify the label.txt to '2 0.3 0.3 0.5 0.5 4 0.3 0.3 0.5 0.5'

but when I detect, each detected box is only classified to one class. Do you know how to fix it?

@KRYSTALLJY are you sure make multi_label=True? or you can debug see every class score.

[glenn-jocher]

@KRYSTALLJY @wzj5133329 multi_label=True is the default setting.

You can clearly see multilabel in action in inference results from detect.py. If you look at an ambiguous object competing classes will appear overlapped with multiple labels for a single box.

[wzj5133329]

@KRYSTALLJY @wzj5133329 multi_label=True is the default setting.

You can clearly see multilabel in action in inference results from detect.py. If you look at an ambiguous object competing classes will appear overlapped with multiple labels for a single box.

@glenn-jocher I can get the desired results，but the class score relatively lower than signal_label in one box,is this normal？

[glenn-jocher]

@wzj5133329 multi-class scores can all vary from 0-1, there is no constraint and there is no softmax applied. Whatever it is you are talking about has nothing to do with multi-label.

[abhigoku10]

@glenn-jocher @wzj5133329 can you please tell me how this multiclass scores work for single bounding box , though same object provided with same bounding box but different lables so that means that we get multiple boxes along with different labels rigth ? @wzj5133329 can you please share your results visualized for reference

[glenn-jocher]

@abhigoku10 multi-label allows one detection to have multiple labels. You can turn it on in NMS here: https://github.com/ultralytics/yolov5/blob/1922ddeac0294a91d585020bbb7290b8e353d813/utils/general.py#L607

[loucif01]

@abhigoku10 multi-label allows one detection to have multiple labels. You can turn it on in NMS here:

https://github.com/ultralytics/yolov5/blob/1922ddeac0294a91d585020bbb7290b8e353d813/utils/general.py#L607

@glenn-jocher multi_label=True, this allows detection to have multiple labels through training or whene executing inference?

[glenn-jocher]

@loucif01 inference. NMS is not used during training.

[loucif01]

@loucif01 inference. NMS is not used during training.

Thank you for your reply! now what i want to know is that does yolo support multi labels bounding boxes? because i want to train yolo on my dataset which contains multilabel bounding boxes (each bounding box can have multiple class labels), so i want to know if yolo accepts this and how to do it? Do i have just repeat the same bounding boxes for multiple labels?

[glenn-jocher]

@loucif01 yes you can simply add additional rows for additional classes with the same box, though our loss function is not specialized for multi-class, so I think there's a side effect of reduced confidence on these that another user discovered.

You do not actually need to train with multi-label labels though, for example the default COCO models are multi-label capable and will display multiple labels for an object (i.e. 0.7 car, 0.6 truck) when multi_label=True.

[loucif01]

@glenn-jocher i did exactly what you said, but unfortunately i didn't get good results. Here is my results:

Class Images Targets P R mAP@.5 mAP@.5:.95: 100% 7/7 [00:06<00:00, 1.16it/s] all 204 1.82e+03 0.342 0.276 0.231 0.0871 CorrosionStain 204 321 0.44 0.302 0.262 0.104 Crack 204 550 0.123 0.0982 0.0365 0.00753 Efflorescence 204 172 0.195 0.151 0.0855 0.0206 ExposedBars 204 356 0.492 0.444 0.391 0.151 Spallation 204 422 0.459 0.386 0.38 0.152

Do you have any suggestion how to improuve it? my dataset have 5 classes : ['CorrosionStain', 'Crack', 'Efflorescence', 'ExposedBars', 'Spallation'], i trained YOLOv5, YOLOR and YOLOX but i still can't get mAP@0.5 > 0.27! Thanks you for your help! PS: I followed this tuto to train my models : https://www.youtube.com/watch?v=MdF6x6ZmLAY He got good results but not me!

[glenn-jocher]

@loucif01 thanks for asking about improving YOLOv5 🚀 training results.

Most of the time good results can be obtained with no changes to the models or training settings, provided your dataset is sufficiently large and well labelled. If at first you don't get good results, there are steps you might be able to take to improve, but we always recommend users first train with all default settings before considering any changes. This helps establish a performance baseline and spot areas for improvement.

If you have questions about your training results we recommend you provide the maximum amount of information possible if you expect a helpful response, including results plots (train losses, val losses, P, R, mAP), PR curve, confusion matrix, training mosaics, test results and dataset statistics images such as labels.png. All of these are located in your project/name directory, typically yolov5/runs/train/exp.

We've put together a full guide for users looking to get the best results on their YOLOv5 trainings below.

Dataset

• Images per class. ≥ 1500 images per class recommended
• Instances per class. ≥ 10000 instances (labeled objects) per class recommended
• Image variety. Must be representative of deployed environment. For real-world use cases we recommend images from different times of day, different seasons, different weather, different lighting, different angles, different sources (scraped online, collected locally, different cameras) etc.
• Label consistency. All instances of all classes in all images must be labelled. Partial labelling will not work.
• Label accuracy. Labels must closely enclose each object. No space should exist between an object and it's bounding box. No objects should be missing a label.
• Background images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.

Model Selection

Larger models like YOLOv5x and YOLOv5x6 will produce better results in nearly all cases, but have more parameters, require more CUDA memory to train, and are slower to run. For mobile deployments we recommend YOLOv5s/m, for cloud deployments we recommend YOLOv5l/x. See our README table for a full comparison of all models.

• Start from Pretrained weights. Recommended for small to medium sized datasets (i.e. VOC, VisDrone, GlobalWheat). Pass the name of the model to the --weights argument. Models download automatically from the latest YOLOv5 release.

  python train.py --data custom.yaml --weights yolov5s.pt
                                           yolov5m.pt
                                           yolov5l.pt
                                           yolov5x.pt
                                           custom_pretrained.pt

• Start from Scratch. Recommended for large datasets (i.e. COCO, Objects365, OIv6). Pass the model architecture yaml you are interested in, along with an empty --weights '' argument:

  python train.py --data custom.yaml --weights '' --cfg yolov5s.yaml
                                                    yolov5m.yaml
                                                    yolov5l.yaml
                                                    yolov5x.yaml

Training Settings

Before modifying anything, first train with default settings to establish a performance baseline. A full list of train.py settings can be found in the train.py argparser.

• Epochs. Start with 300 epochs. If this overfits early then you can reduce epochs. If overfitting does not occur after 300 epochs, train longer, i.e. 600, 1200 etc epochs.
• Image size. COCO trains at native resolution of --img 640, though due to the high amount of small objects in the dataset it can benefit from training at higher resolutions such as --img 1280. If there are many small objects then custom datasets will benefit from training at native or higher resolution. Best inference results are obtained at the same --img as the training was run at, i.e. if you train at --img 1280 you should also test and detect at --img 1280.
• Batch size. Use the largest --batch-size that your hardware allows for. Small batch sizes produce poor batchnorm statistics and should be avoided.
• Hyperparameters. Default hyperparameters are in hyp.scratch.yaml. We recommend you train with default hyperparameters first before thinking of modifying any. In general, increasing augmentation hyperparameters will reduce and delay overfitting, allowing for longer trainings and higher final mAP. Reduction in loss component gain hyperparameters like hyp['obj'] will help reduce overfitting in those specific loss components. For an automated method of optimizing these hyperparameters, see our Hyperparameter Evolution Tutorial.

Further Reading

If you'd like to know more a good place to start is Karpathy's 'Recipe for Training Neural Networks', which has great ideas for training that apply broadly across all ML domains: http://karpathy.github.io/2019/04/25/recipe/

[abhigoku10]

@glenn-jocher @loucif01 thanks for the response,

1. I enabled the multilabel=True in the code and ran the pre-trained coco weights yet I am getting a single box and single labels, can we get multilabel for COCO classes also
   2. I trained on a sample set(20 imgs) that has a single box and multiple labels (ie same bbox values in .txt only class name changed) for few epochs and enabled multilabel = True yet i am not getting single box multiple label but getting multiple boxes similar to detections

Can you let me knw what error i am doing should i change anywhere else . THanks in adavance

[glenn-jocher]

@abhigoku10 as I mentioned above multi-label training is not in place currently.

multi-label inference is automatic if multi_label is true, as I also mentioned above. This is quite obvious in COCO, especially at reduced --conf levels.

[abhigoku10]

@glenn-jocher thanks for the response, I forgot to mention previously I changed the --conf thresh to 0.08,0.05 yet i am not getting the results ie for single bounding box multiple labels. I am doing something wrong in testing this is again for pre-trained coco model

[glenn-jocher]

@abhigoku10 not sure what you mean.

Source image: https://www.purelypetsinsurance.co.uk/media/1138/dog-and-kitten-meeting.jpg

multi_label=False

image 2/3 yolov5/data/images/dog-and-kitten-meeting.jpeg: 448x640 1 cat, 1 dog, Done. (0.174s)

multi_label=True

image 2/3 yolov5/data/images/dog-and-kitten-meeting.jpeg: 448x640 2 cats, 2 dogs, Done. (0.174s)

[PlinyTheMid]

I think the actual problem is trying to train custom dataset with multiple class annotations in a single image. For eg. If there are 3 classes, some images contain only class 1 and the annotation txt file has only annotation pertaining to the class, but if a image has 2 class labels in it, we can only annotate them in the same txt file, to match the image file name.

The Image

The Annotation file

So when we train with a dataset like this, we often get Precision, Recall and mAP scores as absolute 0s. The question is if there's any flag or anything like NMS to enable training with multiple labels in a single image.

[nikitabelooussovbtis]

What would you recommend for making the labels more readable when multilabel is enabled. Once one object has more than two boxes is becomes difficult to read.

[ma-aquifi]

Can i follow a similar procedure for multi labeling in YOLOv7 ?

[glenn-jocher]

@ma-aquifi For multi-labeling with YOLOv7, you can follow a similar procedure as with YOLOv5. First, ensure that you have the multi-label flag enabled in the configuration. Then annotate your images and create label files that include multiple class labels for the same bounding box.

When working with multiple labels in a single image, it's important to format the labels in a way that makes them readable and understandable. One approach is to have each class label on a separate line in the label file, with the corresponding bounding box coordinates. Additionally, you can consider using different colors or annotations to visually distinguish between the different labels in the image.

As for YOLOv7, please note that YOLOv7 is not an official release from the original YOLO creators. The original YOLOv7 was developed by the Ultralytics team and is based on YOLOv5, with various enhancements and improvements. Therefore, you can apply similar multi-label training techniques in YOLOv7 as well.

I recommend referring to the official documentation and forums for YOLOv7 for specific guidance on multi-label training and any additional features or changes introduced in that version.