Model Ensembling Tutorial

[glenn-jocher]

📚 This guide explains how to use YOLOv5 🚀 model ensembling during testing and inference for improved mAP and Recall. UPDATED 25 September 2022.

From https://www.sciencedirect.com/topics/computer-science/ensemble-modeling:

Ensemble modeling is a process where multiple diverse models are created to predict an outcome, either by using many different modeling algorithms or using different training data sets. The ensemble model then aggregates the prediction of each base model and results in once final prediction for the unseen data. The motivation for using ensemble models is to reduce the generalization error of the prediction. As long as the base models are diverse and independent, the prediction error of the model decreases when the ensemble approach is used. The approach seeks the wisdom of crowds in making a prediction. Even though the ensemble model has multiple base models within the model, it acts and performs as a single model.

Before You Start

Clone repo and install requirements.txt in a Python>=3.7.0 environment, including PyTorch>=1.7. Models and datasets download automatically from the latest YOLOv5 release.

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Test Normally

Before ensembling we want to establish the baseline performance of a single model. This command tests YOLOv5x on COCO val2017 at image size 640 pixels. yolov5x.pt is the largest and most accurate model available. Other options are yolov5s.pt, yolov5m.pt and yolov5l.pt, or you own checkpoint from training a custom dataset ./weights/best.pt. For details on all available models please see our README table.

$ python val.py --weights yolov5x.pt --data coco.yaml --img 640 --half

Output:

val: data=./data/coco.yaml, weights=['yolov5x.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.65, task=val, device=, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=True, project=runs/val, name=exp, exist_ok=False, half=True
YOLOv5 🚀 v5.0-267-g6a3ee7c torch 1.9.0+cu102 CUDA:0 (Tesla P100-PCIE-16GB, 16280.875MB)

Fusing layers... 
Model Summary: 476 layers, 87730285 parameters, 0 gradients

val: Scanning '../datasets/coco/val2017' images and labels...4952 found, 48 missing, 0 empty, 0 corrupted: 100% 5000/5000 [00:01<00:00, 2846.03it/s]
val: New cache created: ../datasets/coco/val2017.cache
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 157/157 [02:30<00:00,  1.05it/s]
                 all       5000      36335      0.746      0.626       0.68       0.49
Speed: 0.1ms pre-process, 22.4ms inference, 1.4ms NMS per image at shape (32, 3, 640, 640)  # <--- baseline speed

Evaluating pycocotools mAP... saving runs/val/exp/yolov5x_predictions.json...
...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.504  # <--- baseline mAP
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.688
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.546
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.351
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.551
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.382
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.628
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.681  # <--- baseline mAR
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.524
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.735
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.826

Ensemble Test

Multiple pretraind models may be ensembled togethor at test and inference time by simply appending extra models to the --weights argument in any existing val.py or detect.py command. This example tests an ensemble of 2 models togethor:

• YOLOv5x
• YOLOv5l6

python val.py --weights yolov5x.pt yolov5l6.pt --data coco.yaml --img 640 --half

Output:

val: data=./data/coco.yaml, weights=['yolov5x.pt', 'yolov5l6.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=True, project=runs/val, name=exp, exist_ok=False, half=True
YOLOv5 🚀 v5.0-267-g6a3ee7c torch 1.9.0+cu102 CUDA:0 (Tesla P100-PCIE-16GB, 16280.875MB)

Fusing layers... 
Model Summary: 476 layers, 87730285 parameters, 0 gradients  # Model 1
Fusing layers... 
Model Summary: 501 layers, 77218620 parameters, 0 gradients  # Model 2
Ensemble created with ['yolov5x.pt', 'yolov5l6.pt']  # Ensemble notice

val: Scanning '../datasets/coco/val2017.cache' images and labels... 4952 found, 48 missing, 0 empty, 0 corrupted: 100% 5000/5000 [00:00<00:00, 49695545.02it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 157/157 [03:58<00:00,  1.52s/it]
                 all       5000      36335      0.747      0.637      0.692      0.502
Speed: 0.1ms pre-process, 39.5ms inference, 2.0ms NMS per image at shape (32, 3, 640, 640)  # <--- ensemble speed

Evaluating pycocotools mAP... saving runs/val/exp3/yolov5x_predictions.json...
...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.515  # <--- ensemble mAP
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.699
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.557
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.356
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.563
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.387
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.638
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.689  # <--- ensemble mAR
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.526
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.743
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.844

Ensemble Inference

Append extra models to the --weights argument to run ensemble inference:

python detect.py --weights yolov5x.pt yolov5l6.pt --img 640 --source data/images

Output:

detect: weights=['yolov5x.pt', 'yolov5l6.pt'], source=data/images, imgsz=640, conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False
YOLOv5 🚀 v5.0-267-g6a3ee7c torch 1.9.0+cu102 CUDA:0 (Tesla P100-PCIE-16GB, 16280.875MB)

Fusing layers... 
Model Summary: 476 layers, 87730285 parameters, 0 gradients
Fusing layers... 
Model Summary: 501 layers, 77218620 parameters, 0 gradients
Ensemble created with ['yolov5x.pt', 'yolov5l6.pt']

image 1/2 /content/yolov5/data/images/bus.jpg: 640x512 4 persons, 1 bus, 1 tie, Done. (0.063s)
image 2/2 /content/yolov5/data/images/zidane.jpg: 384x640 3 persons, 2 ties, Done. (0.056s)
Results saved to runs/detect/exp2
Done. (0.223s)

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@silenus092 currently ensembling is only available in val.py and detect.py:

python detect.py --weights yolov5m6.p5 yolov5l6.pt

[JongWooBAE]

I have some questions When ensembling two models, model's classes must be same? I want to use two models with different classes. But I can't find how to do.

[glenn-jocher]

@JongWooBAE yes classes must be identical.

[davidbroberts]

@glenn-jocher, is this functionality available when loading from torch hub?

[glenn-jocher]

@davidbroberts no, currently torch hub can only load one model per command, though you can load multiple models into a workspace at the same time, i.e.:

yolov5s = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # load one model
yolov5m = torch.hub.load('ultralytics/yolov5', 'yolov5m')  # load a second model

[davidbroberts]

That's what I thought. Thank you for all the great work!

[pj991207]

I have some questions. I want to know the proper number of models when ensembeling pre-trained models(ex, yolov5x.pt, yolov5l6.pt etc...)

[Runist]

How to ensemble models with different input sizes

[myasser63]

@glenn-jocher Can it be done with two models trained on different datasets?

[Runist]

你好，您的邮件我已收到。我将在近期查看，尽快给你回复。

[glenn-jocher]

@myasser63 model ensembling is intended for multiple models trained on the same dataset.

You can train one model on multiple datasets though: https://community.ultralytics.com/t/how-to-combine-weights-to-detect-from-multiple-datasets

[myasser63]

@glenn-jocher Thanks for your help. It's very helpful

[kimkihoon0515]

@glenn-jocher I'm trying to test my trained model by using torch like this => model = torch.hub.load('./utils','custom',path='/opt/ml/yolov5/runs/train/exp/weights/best.pt',source='local') is there any way to ensemble more than 2 models ??

[glenn-jocher]

@kimkihoon0515 model ensembling is available with detect.py and val.py. PyTorch Hub will load individual models for you given your command.

[kimkihoon0515]

@glenn-jocher oic btw tensor.split() now works pretty well on torch ver 1.7.1 also thx for that :)

[arunmack789]

how to detect images with two model first model is yolov5s model(pretrained model) for human detetcion and custom model for detect trolley @glenn-jocher

[glenn-jocher]

@arunmack789 👋 Hello! Thanks for asking about handling inference results. YOLOv5 🚀 PyTorch Hub models allow for simple model loading and inference in a pure python environment without using detect.py.

Simple Inference Example

This example loads a pretrained YOLOv5s model from PyTorch Hub as model and passes an image for inference. 'yolov5s' is the YOLOv5 'small' model. For details on all available models please see the README. Custom models can also be loaded, including custom trained PyTorch models and their exported variants, i.e. ONNX, TensorRT, TensorFlow, OpenVINO YOLOv5 models.

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # or yolov5m, yolov5l, yolov5x, etc.
# model = torch.hub.load('ultralytics/yolov5', 'custom', 'path/to/best.pt')  # custom trained model

# Images
im = 'https://ultralytics.com/images/zidane.jpg'  # or file, Path, URL, PIL, OpenCV, numpy, list

# Inference
results = model(im)

# Results
results.print()  # or .show(), .save(), .crop(), .pandas(), etc.

results.xyxy[0]  # im predictions (tensor)
results.pandas().xyxy[0]  # im predictions (pandas)
#      xmin    ymin    xmax   ymax  confidence  class    name
# 0  749.50   43.50  1148.0  704.5    0.874023      0  person
# 2  114.75  195.75  1095.0  708.0    0.624512      0  person
# 3  986.00  304.00  1028.0  420.0    0.286865     27     tie

See YOLOv5 PyTorch Hub Tutorial for details.

Good luck 🍀 and let us know if you have any other questions!

[arunmack789]

not working my problem i used pretrained model yolovs.pt and custom trained model but it gets size error problem Sizes of tensors must match except in dimension 1. yolov5 @glenn-jocher

[glenn-jocher]

@arunmack789 ensembling is restricted to models trained on the same dataset, your problem is due to user error.

[myasser63]

@glenn-jocher is it possible to ensemble YOLOv5s with other models other than YOLOv5

[glenn-jocher]

@myasser63 sure, any custom models trained with this repo should work. i.e. you can also go grab yolov3.pt from https://github.com/ultralytics/yolov3 and ensemble it with YOLOv5 models from this repo.

[myasser63]

@glenn-jocher Can the ensemble be done using onnx?

[Runist]

你好，您的邮件我已收到。我将在近期查看，尽快给你回复。

[glenn-jocher]

@myasser63 no

[akshay-ast]

Can I ensemble the model with wts that are trained on different classes?

[glenn-jocher]

@akshay-ast no

[AlexofNTU]

What is the mechanism behind ensemble in YOLOv5?

I just try this example and found an interesting result which is not easy to be explained!

In the following image, I tried to perform detection using yolov5I6, yolov5x, and yolov5I6 + yolov5x on the same image. In the left image, the leftmost truncated person is detected with an imprecise bounding box (probability=0.58) which tried to enclose the unseen head. In the center image, the bounding box of this person is more precise and the probability is 0.71. However, in the right image, the bounding box corresponding to the same person has 0.80 probability which is higher than the ones, 0.58 & 0.71 respectively, in the left and center images.

I can't figure out why because if we put these detection results of the same truncated person in the left and right images together, the one from the left image should be eliminated by the one in the center image. However, after trying to trace how NMS works in YOLOv5, I can't find any script tried to increase the probability after merging.

Besides, for the detected bus, I also can't understand the ensemble result. The bus in the right image has probability of 0.92. However, since the same bus in the center image has higher probability, the merged result should have a probability of 0.92 instead of 0.93.

Could anyone explain why the results are counter-intuitive?

[glenn-jocher]

@AlexofNTU the YOLOv5 Ensemble() module is hard coded to an NMS ensembling method. All model outputs are concatenated before being passed to NMS. The other two methods are possible when all models share the same size outputs.

https://github.com/ultralytics/yolov5/blob/5774a1514de193c74ecc5203281da8de3c13f9af/models/experimental.py#L61-L72

[AlexofNTU]

@AlexofNTU the YOLOv5 Ensemble() module is hard coded to an NMS ensembling method. All model outputs are concatenated before being passed to NMS. The other two methods are possible when all models share the same size outputs.

https://github.com/ultralytics/yolov5/blob/5774a1514de193c74ecc5203281da8de3c13f9af/models/experimental.py#L61-L72

Thanks for the reply! Even though outputs from different models are concatenated before NMS, I still don't understand why the probability of the survived bounding boxes is possible to be increased. In the NMS function, the survived bounding box has the updated bounding boxes because they are the averaged results of the eliminated bounding boxes but the probability doesn't!

[glenn-jocher]

It's interesting indeed. I can't state the cause with any certainty.

[brunopatricio2012]

It is possible to join two trained frozen models into a single model with different classes?

[glenn-jocher]

@brunopatricio2012 no, you can't ensemble two models trained on different datasets, but you can train one model on two different datasets. See https://community.ultralytics.com/t/how-to-combine-weights-to-detect-from-multiple-datasets

[moahaimen]

hi I am working on object detection using Yolov5 with custom dataset I designed i have several problems i wish i can contact someone and send him my errors in object detection directly , it always give other object name or it give little confidence sometimes it give 2 bound boxes for the same object please help

[moahaimen]

@Zzh-tju ensembling runs multiple models, while TTA tests a single model at with different augmentations. Typically I've seen the best result when merging output grids directly, (i.e. ensembling YOLOv5l and YOLOv5x), rather than simply appending boxes from multiple models for NMS to sort out. This is not always possible however, for example Ensembling an EfficientDet model with YOLOv5x, you can not merge grids, you must use NMS or WBF (or Merge NMS) to get a final result.

i need to make 3 types of yolov5 models in Ensembled Deep Learning so The aggregation voting will give the final decision how can i do that ?

[Thanapong-khajon]

@glenn-jocher Do you have any emsemble method with faster r-cnn and yolov5? i trained faster r-cnn with tensorflow and i got model file .pb . Should a faster r-cnn be convert from .pb to .pt? or Something?

[glenn-jocher]

@Thanapong-khajon Ensemble() only works with PyTorch models, but yes if you have a PyTorch Faster RCNN model it might work.

[Thanapong-khajon]

@Thanapong-khajon Ensemble() only works with PyTorch models, but yes if you have a PyTorch Faster RCNN model it might work.

Do you have weights of Faster RCNN ?

[pathikg]

I want to ensemble two yolov5x6 models trained on the same data with some variation I saw the Ensemble() class mentioned in above issues but I was a bit confused on how to implement it in a python script

In other words, how do I exactly use that Ensemble in order to create an ensemble of my 2 models within a python script/function just by passing the models?

[glenn-jocher]

@pathikg YOLOv5 ensembling is automatically built into detect.py and val.py, so simply pass two weights:

python detect.py --weights yolov5s.pt yolov5m.pt

[pathikg]

@pathikg YOLOv5 ensembling is automatically built into detect.py and val.py, so simply pass two weights:

python detect.py --weights yolov5s.pt yolov5m.pt

Thanks @glenn-jocher for quick reply but I want to do this in a python script At present, I am loading model from torch hub with my custom weights and then doing the inference on respective images. At present I've two such models, and I want to make ensemble of the same so is there any way I can do that as well?

[glenn-jocher]

I’d follow the code in detect.py and use the Ensemble() module from models/common.py

[pathikg]

You mean from models/experimental.py? cuz there's no Ensemble() module present in common.py :/

[glenn-jocher]

yes sorry in experimental.py

[AfiqueAye]

@glenn-jocher I really need your help for one of my problems. I have a dataset trained on cat, dog and pen, and after training, I have got dataset1.pt best file Now, I trained another model with new data, for example I have taken just images of cow, and got dataset2.pt best file. Both of the trained model can detect the images separately. But I want to make them one model, so that it will detect all the images (cat, dog, pen and cow) using a single weight file. Can I do it using ensemble techniques? will this work? python detect.py --weights dataset1.pt dataset2.pt --img 640 --source data/images/cat or how can I do this, is there any way, rather than creating a new dataset will all the images, and trained again. Please reply, thanks

[michael-mayo]

@Zzh-tju ensembling and TTA are not mutually exclusive. You can TTA a single model, and you can ensemble a group of models with or without TTA:

python detect.py --weights model1.pt model2.pt --augment

Hi @glenn-jocher , what would be the best practice for deploying an ensemble model like this? I know we can export the individual models for different deployment frameworks, but how would I export an ensemble?

[arbaz-pivotchain]

Hello, I read a previous discussion about ensembling multiple trained networks by simply passing more than 1 weight file during inference.

My question is what technique is being used for fusing the predictions? Is it majority voting on the bounding boxes? Some weighted averaging?

[michael-mayo]

I ended up predicting each image with all of the ensemble members separately, and then combining the bounding box predictions together and doing a second stage NMS to generate a final combined prediction. Seems to work OK. Another possibility is to average the weights of the ensemble members into an averaged model like they do in federated learning, but I have not properly evaluated that method yet.

[glenn-jocher]

@michael-mayo that's a good approach! Typically, the ensembling technique involves averaging the model weights and biases instead of the predictions. Here, you are combining the predictions which can be achieved using the NMS algorithm. Keep in mind that it's important to experiment and choose the best method based on the specific problem and the performance of each approach.

[michael-mayo]

I should also add that for each ensemble member I trained using a different global random seed, and a different (5/6) subset of the training data, to improve ensemble diversity.

[glenn-jocher]

@michael-mayo That's a great technique to improve ensemble diversity. It can help reducing the chances of overfitting (which can happen if all ensemble members are trained on exactly the same data) and increase the robustness of the final predictions.