CUSTOM TRAINING EXAMPLE (OLD)

[glenn-jocher]

This guide explains how to train your own custom dataset with YOLOv3.

Before You Start

Clone this repo, download COCO dataset, and install requirements.txt dependencies, including Python>=3.7 and PyTorch>=1.4.

git clone https://github.com/ultralytics/yolov3
bash yolov3/data/get_coco2017.sh  # 19GB
cd yolov3
pip install -U -r requirements.txt

Train On Custom Data

1. Label your data in Darknet format. After using a tool like Labelbox to label your images, you'll need to export your data to darknet format. Your data should follow the example created by get_coco2017.sh, with images and labels in separate parallel folders, and one label file per image (if no objects in image, no label file is required). The label file specifications are:

• One row per object
• Each row is class x_center y_center width height format.
• Box coordinates must be in normalized xywh format (from 0 - 1). If your boxes are in pixels, divide x_center and width by image width, and y_center and height by image height.
• Class numbers are zero-indexed (start from 0).

Each image's label file must be locatable by simply replacing /images/*.jpg with /labels/*.txt in its pathname. An example image and label pair would be:

../coco/images/train2017/000000109622.jpg  # image
../coco/labels/train2017/000000109622.txt  # label

An example label file with 5 persons (all class 0):

*2. Create train and test `.txtfiles.** Here we createdata/coco16.txt, which contains the first 16 images of the COCO2017 dataset. We will use this small dataset for both training and testing. Each row contains a path to an image, and remember one label must also exist in a corresponding/labels` folder for each image containing objects.

*3. Create new `.namesfile** listing the class names in our dataset. Here we use the existingdata/coco.namesfile. Classes are **zero indexed**, sopersonis class0,bicycleis class1`, etc.

*4. Create new `.datafile** with your class count (COCO has 80 classes), paths to train and validation datasets (we use the same images twice here, but in practice you'll want to validate your results on a separate set of images), and with the path to your*.namesfile. Save asdata/coco16.data`.

5. Update yolov3-spp.cfg (optional). By default each YOLO layer has 255 outputs: 85 values per anchor [4 box coordinates + 1 object confidence + 80 class confidences], times 3 anchors. Update the settings to filters=[5 + n] * 3 and classes=n, where n is your class count. This modification should be made in all 3 YOLO layers.

6. (OPTIONAL) Update hyperparameters such as LR, LR scheduler, optimizer, augmentation settings, multi_scale settings, etc in train.py for your particular task. If in doubt about these settings, we recommend you start with all-default settings before changing anything.

7. Train. Run python3 train.py --cfg yolov3-spp.cfg --data data/coco16.data --nosave to train using your custom .data and .cfg. By default pretrained --weights yolov3-spp-ultralytics.pt is used to initialize your model. You can instead train from scratch with --weights '', or from any other weights or backbone of your choice, as long as it corresponds to your *.cfg.

Visualize Results

Run from utils import utils; utils.plot_results() to see your training losses and performance metrics vs epoch. If you don't see acceptable performance, try hyperparameter tuning and re-training. Multiple results.txt files are overlaid automatically to compare performance.

Here we see training results from data/coco64.data starting from scratch, a darknet53 backbone, and our yolov3-spp-ultralytics.pt pretrained weights.

Run inference with your trained model by copying an image to data/samples folder and running
python3 detect.py --weights weights/last.pt

Reproduce Our Results

To reproduce this tutorial, simply run the following code. This trains all the various tutorials, saves each results*.txt file separately, and plots them together as results.png. It all takes less than 30 minutes on a 2080Ti.

git clone https://github.com/ultralytics/yolov3
python3 -c "from yolov3.utils.google_utils import gdrive_download; gdrive_download('1h0Id-7GUyuAmyc9Pwo2c3IZ17uExPvOA','coco2017demos.zip')"  # datasets (20 Mb)
cd yolov3
python3 train.py --data coco64.data --batch 16 --epochs 300 --nosave --cache --weights '' --name from_scratch
python3 train.py --data coco64.data --batch 16 --epochs 300 --nosave --cache --weights yolov3-spp-ultralytics.pt --name from_yolov3-spp-ultralytics
python3 train.py --data coco64.data --batch 16 --epochs 300 --nosave --cache --weights darknet53.conv.74 --name from_darknet53.conv.74
python3 train.py --data coco1.data --batch 1 --epochs 300 --nosave --cache --weights darknet53.conv.74 --name 1img
python3 train.py --data coco1cls.data --batch 16 --epochs 300 --nosave --cache --weights darknet53.conv.74 --cfg yolov3-spp-1cls.cfg --name 1cls

Reproduce Our Environment

To access an up-to-date working environment (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled), consider a:

• GCP Deep Learning VM with $300 free credit offer: See our GCP Quickstart Guide
• Google Colab Notebook with 12 hours of free GPU time: Google Colab Notebook
• Docker Image from https://hub.docker.com/r/ultralytics/yolov3. See Docker Quickstart Guide

[glenn-jocher]

@Hzyu810225 this is already the default action if you don’t specify weights.

[coolmarat]

Thank you for sharing. I successfully train single-class net. But now I try to train on multiple classses and get error:

Please help me resolve this.

[glenn-jocher]

@coolmarat your cfg may not be configured correctly. Make sure these are both updated to correctly reflect your class count:

[coolmarat]

@glenn-jocher , I have 13 classes and set filters to 15+3*13=54. I replace all strings "filters=255" with "filters=54".

At first I forget to change classes count, but there was another error with message "model classes exceed total count", when I set correct classes count it disappeared. But now I have this error...

[glenn-jocher]

Well make sure you are using the right cfg file and do some debugging. The error says that its trying to reshape something into the vector shown, which clearly won't work:

2292032/13/13/32/3
Out[8]: 141.2741617357002

Another option if you have <= 80 classes is just the use the default cfg.

[glenn-jocher]

ah I had the compute wrong, it is looking for size 18 in that dimension, not 15:

292032/13/13/32/3
Out[9]: 18.0

[coolmarat]

@glenn-jocher. I use tiny.cfg, because standard yolov3.cfg try to obtain all my 16GB RAM. Can it be problem place? I found this issue https://github.com/ultralytics/yolov3/issues/415 and download weights, but still no luck

[coolmarat]

Can anybody share yolov3-tiny.cfg file for exact 13 classes detection pls?

[oscarzasa]

Hi @glenn-jocher,

In the results graphs, why do the values of Precision and F1 drop at the end of the training?

[mkmk001]

I Manage to launch on GPU. But now I have the following error :

hi, i have met the same error with you.did you solved that?

[jveitchmichaelis]

@oscarzasa because the pipeline drops the confidence threshold almost to zero at final test time. I'm not convinced this is a good way to benchmark models, but it does give an improvement on mAP. Essentially you're "hacking" the fact that mAP doesn't care if some of your bounding boxes are rubbish. So when you have a low threshold, sometimes you might get some very low confidence boxes appearing that match with the correct prediction. This is why during training mAP is evaluated at a threshold of 0.1, mAP tends to be lower before spiking at the end.

This mainly affects precision because you're generating more boxes and it's a measure of how many correct predictions you made out of the number of total predictions. Recall isn't affected because it measures how well you recover the ground truth (which you will do anyway for a good model, regardless of the number of spurious boxes - this is where mAP isn't good). F1 is the harmonic mean of the two, so it goes down.

It's worth noting that when you compare results to other libraries like darknet, they validate mAP on a much higher confidence threshold (darknet is 0.25 for example).

This has much less of an effect on "good" models, i.e. ones which are well regularised and don't give spurious results, but it may artificially boost benchmark performance on a bad model.

You can look at other metrics like https://arxiv.org/abs/1807.01696 as well as mAP. It might also be improved with a better NMS threshold. What you want is for lots of the overlapping boxes to be suppressed and to keep the low confidence ones that cover classes you missed.

This is fine for benchmarking COCO (and as you can see from the models, they work just fine), but in reality if you're training on your own dataset, I would suggest setting the test confidence threshold to be something realistic for your application - for example, would you accept the prediction of a classifier if it was only 20% confident? 5%? 0.1%?

[jveitchmichaelis]

@glenn-jocher is there any intuition behind setting the gains in the loss function?

I've trained a custom model with 5 classes. I got pretty good results using the 80 class config (gets to about 55mAP which is roughly what I'd expect). For interest, my results plateau at about 60 epochs without putting more effort into augmentation or adjusting learning rates. Typically I was seeing passable results after the first epoch.

I'm not training with an adjusted config file, still using darknet53.conv.74 so I assume the output YOLO layers are randomly initialised. It seems to train OK, although it takes a few epochs for anything sensible to start appearing. For now I just multiplied cls_gain and obj_gain by 5/80. I guess I can evolve them to improve, but just wondering how they relate to the number of output classes (or the distribution within those classes)

[coolmarat]

Hello guys!

I want to complete my source set of image during training. Is it possible to train model with one set of images for some epochs then resume training on other set of images? Will it improve performance?

[glenn-jocher]

@coolmarat no, you should train with all available images at all times.

@jveitchmichaelis ok. 5 classes should train fine with yolov3-spp.cfg, but you'll probably get better mAP if you create your own yolov3-spp-5cls.cfg file. Beware that default training is 273 epochs, with LR drops at 80 and 90% epochs (about 220 and 240), so if you want to stop at 50 you should train from the start with --epochs 50. But then check results.png to see if your validation losses are actually plateaued or rising after 50 epochs, and if not then you should train longer. You can get faster results (but maybe not better results) just using yolov3-spp.weights as the backbone instead of darknet53.conv.74. Here are the 4 backbone options on the coco64_img.datasets.

python3 train.py --data data/coco_64img.data --batch-size 16 --accumulate 1 --nosave --weights weights/yolov3-spp.weights --transfer --name yolov3-spp_transfer  # TRANSFER LEARNING COMPARISON
python3 train.py --data data/coco_64img.data --batch-size 16 --accumulate 1 --nosave --weights '' --name from_scratch
python3 train.py --data data/coco_64img.data --batch-size 16 --accumulate 1 --nosave --weights weights/darknet53.conv.74 --name darknet53_backbone
python3 train.py --data data/coco_64img.data --batch-size 16 --accumulate 1 --nosave --weights weights/yolov3-spp.weights --name yolov3-spp_backbone

[jveitchmichaelis]

Cheers @glenn-jocher - I already have a custom config file and I'm training for 75, seems to do the trick. Somehow I had managed to write over my darknet weights with freshly initialised values, so everything was training from scratch.

Still, the question about the gain hyperparameters - I guess these are tuned on COCO, so does it make a difference how many classes you use? For example the comment in the code says:

'obj': 21.35, # obj loss gain (*=80 for uBCE with 80 classes) which suggests it should be modified if you're not using COCO, or is that only if you're using uBCE?

[glenn-jocher]

@jveitchmichaelis ah, uBCE is a different type of architecture we experimented with that combined detection and classification into the same vector. It worked better with different hyps, but I wouldn't worry about that. The current hyps work best with all default settings, they were determined via a genetic algorithm on partially trained COCO at img-size 320.

You can evolve your own hyperparameters also, see https://github.com/ultralytics/yolov3/issues/392.

[glenn-jocher]

@jveitchmichaelis I was thinking about the topic a bit more. If I compare the latest COCO trainings at 416 and 608, the only main difference in the losses is the obj loss. Orange is 416 and blue is 608. The hyperparameters were evolved at 320, so it makes sense that we might want to scale hyp['obj'] with the img_size. According to the plots below, a rough scaling might be hyp['obj']*=img_size/320, in order to maintain the same loss magnitude at the higher img-sizes.

[coolmarat]

Sometimes my trained model gives false-positive detections. Can I improve it by adding images with this error-detected objects with empty txt?

[glenn-jocher]

@coolmarat of course. If you are training on detecting certain objects, you generally want to supply 'background' images in the training (and test) set as well in a 1:1 object_img:background_img ratio to help the model learn what is not an object. COCO images are good for supplying these backgrounds.

[smalldroid]

@glenn-jocher I try to train yolov3 to detect some small object such as gun, knife on security camera with image size 1920x1080. When training yolov3 model, Does i need resize image extract from that camera to 416 to help model learning feature from image? Thank you!

[glenn-jocher]

@smalldroid resizing is handled automatically to the --img-size you specify.

[smalldroid]

@smalldroid resizing is handled automatically to the --img-size you specify.

Thank you. All image was trained with --img-size default value = 416. My result is greenline in below image. Does I need train more epochs to improve recall. It seems my model detects gun in 1920x1080 image with bad result. Gun object have small size in image 1920x1080. Is there any way to improve this? Should i try yolov3-spp version? Thank you!

[glenn-jocher]

@smalldroid, yes there's always lots of ways to improve results! You can certainly try spp, higher resolution training, multi-scale, hyperparameter evolution, more aggressive augmentation, etc. If you need help, we offer consulting in this area, from providing expert advice on your specific task up to delivery of fully customized, end-to-end AI solutions for our clients. We'd be happy to discuss your problem with you in more detail! https://www.ultralytics.com/

[FranciscoReveriano]

@smalldroid resizing is handled automatically to the --img-size you specify.

Thank you. All image was trained with --img-size default value = 416. My result is greenline in below image. Does I need train more epochs to improve recall. It seems my model detects gun in 1920x1080 image with bad result. Gun object have small size in image 1920x1080. Is there any way to improve this? Should i try yolov3-spp version? Thank you!

From experience. It seems that larger images do better when trained at a nearer size to the original size. I would advice that you lower the batch size and try to train as close as possible to your original size. You will see dramatically different results.

[smalldroid]

Thanks for your answers. I tried to train with spp model and recall increased 0.7->0.8. Increasing input-size improves detection too. I will try to train with higher input-size with lower batch size. My Pc only has 1 VGA RTX2080ti with 11GB. I will try my best.

[dakdouky]

Hi @glenn-jocher I'm wondering why my curves on this tutorial are pretty similar to yours but with different values?

[glenn-jocher]

@MOHAMEDELDAKDOUKY that's fine they look good. The repo is always undergoing changes/improvements.

[PointCloudNiphon]

I have get a coco2014 dataset from the office website, So I can transfer the json annotations to single txts as you required?

[dakdouky]

@PointCloudNiphon Yes, you will have to parse these .json files and convert them to the format specified above.

[LIUhansen]

I did not perform the third step which is DOWNLOAD COCO，due to outline from China. And when I train own data, it can run a while, then occur an error, FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/home/zangxh/Sen/yolo']

[glenn-jocher]

@LIUhansen is there a Google Drive equivalent we can use for China? This seems to be a common problem.

[LIUhansen]

Baidu Cloud. But,It seems that third step could be skip. My error caused by no space left on device.

[MayurDhanaraj]

Hello,

Many thanks for sharing this amazing repo!

I followed instructions for custom data training and when I run train.py, I get the following error: `Namespace(accumulate=4, adam=False, arc='default', batch_size=16, bucket='', cache_images=False, cfg='cfg/yolov3.cfg', data='data/vedai_train.data', device='', epochs=273, evolve=False, img_size=416, multi_scale=False, name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, transfer=False, var=None, weights='weights/ultralytics68.pt') Using CUDA device0 _CudaDeviceProperties(name='GeForce RTX 2060', total_memory=6144MB)

Traceback (most recent call last): File "train.py", line 462, in train() # train normally File "train.py", line 111, in train chkpt = torch.load(weights, map_location=device) File "C:\Users\mayur\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 426, in load return _load(f, map_location, pickle_module, pickle_load_args) File "C:\Users\mayur\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 603, in _load magic_number = pickle_module.load(f, pickle_load_args) _pickle.UnpicklingError: invalid load key, '<'.`

Could you please let me know how to rectify this and continue training ?

[glenn-jocher]

@MayurDhanaraj I'm not sure. It's possible your downloaded weights are corrupted or your dependencies are not correct. Try recloning a fresh copy or using one of our working environments: https://github.com/ultralytics/yolov3#reproduce-our-environment

[MayurDhanaraj]

Thank you for the quick response @glenn-jocher. I am new to GitHub so this question may be silly, but I follow the step by step instructions to train on custom data and the instructions did not mention downloading weights from the official yolov3 webpage. Am I required to do it and place the weights in a specific path ? Please let me know.

[glenn-jocher]

@MayurDhanaraj this repo will attempt to download any --weights you request automatically, i.e. --weights yolov3-spp.pt, or you can specify randomly initialized weights with --weights ''.

[pushgct]

Im training custom dataset in colab. I get following error:

!python3 train.py --data data/bdd100k.data --weights yolov3.weights --cfg cfg/yolov3-bdd100k.cfg

Namespace(accumulate=4, adam=False, arc='default', batch_size=16, bucket='', cache_images=False, cfg='cfg/yolov3-bdd100k.cfg', data='data/bdd100k.data', device='', epochs=273, evolve=False, img_size=416, multi_scale=False, name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, var=None, weights='yolov3.weights') Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Caching labels: 0% 0/32 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 446, in train() # train normally File "train.py", line 189, in train cache_images=opt.cache_images and not opt.prebias) File "/content/drive/My Drive/darknet/yolov3/utils/datasets.py", line 335, in init assert l.shape[1] == 5, '> 5 label columns: %s' % file AssertionError: > 5 label columns: ./labels/b1c66a42-6f7d68ca.txt

My images are in yolov3/images/.jpg directory labels inside yolov3/labels/.txt directory. What Am I doing wrong here?

[glenn-jocher]

@pushgct as the assertion error states, your labels do not have 5 columns. Your labels need to be in class x_center y_center width height format. See the COCO labels and follow that format.

[pushgct]

I have now the correct label format in each .txt files: 0 0.390234375 0.2965277777777778 0.39765625 0.3125 0 0.378515625 0.3090277777777778 0.3859375 0.34444444444444444 0 0.355859375 0.31180555555555556 0.3734375 0.35555555555555557 0 0.341796875 0.3194444444444444 0.36484375 0.3680555555555556 0 0.3109375 0.33055555555555555 0.33515625 0.3875 0 0.2265625 0.33125 0.29921875 0.4305555555555556

could you help me with structure of labels and images directory? I get following error from Google Colab. I have my images and labels in following directories though: My images are in yolov3/images/.jpg directory labels inside yolov3/labels/.txt directory.

contents of train.txt: images/00abd8a7-ecd6fc56.jpg images/0c1d07e3-8cb61d13.jpg images/00d9e313-926b6698.jpg images/000d35d3-41990aa4.jpg images/00e5e793-22614772.jpg images/0ed0d326-5ec5573f.jpg

!python3 train.py --data data/bdd100k.data --weights yolov3.weights --cfg cfg/yolov3-bdd100k.cfg

Namespace(accumulate=4, adam=False, arc='default', batch_size=16, bucket='', cache_images=False, cfg='cfg/yolov3-bdd100k.cfg', data='data/bdd100k.data', device='', epochs=273, evolve=False, img_size=416, multi_scale=False, name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, var=None, weights='yolov3.weights') Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Caching labels: 100% 56/56 [00:00<00:00, 27535.88it/s] Traceback (most recent call last): File "train.py", line 446, in train() # train normally File "train.py", line 189, in train cache_images=opt.cache_images and not opt.prebias) File "/content/drive/My Drive/darknet/yolov3/utils/datasets.py", line 380, in init assert nf > 0, 'No labels found. See %s' % help_url

[gabrielebaris]

Hi, correct me if I am wrong but it seems that the --transfer flag is no more among the possible flags of train.py. Is it right? If yes, is there an easy way to have the same result? Thanks

[glenn-jocher]

@gabrielebaris yes we removed the --transfer flag. To start training from an existing set of weights simply use --weights --cfg, i.e.:

python3 train.py --weights yolov3-spp-ultralytics.pt --cfg yolov3-spp.cfg --data yourdatahere

To train from scratch:

python3 train.py --weights '' --cfg yolov3-spp.cfg --data yourdatahere

[akshu281]

Hello,

I have a set of different bag images to be detected which is very small considering the whole image's resolution. Since this class is already part of COCO class, I am planning to fine-tune my custom dataset using the COCO weights. Will the below command help me achieve the same ??

python3 train.py --weights ultralytics68.pt --cfg yolov3-spp.cfg --data yourdatahere

I have prepared my object.names file, .yolov3-spp.cfg file to reflect classes=1 and filters=18 in 3 places and a custom .data file specifying the classes and path. Is this all the changes that is needed to do a transfer learning ? I doubt as these steps are very similar to the instructions to train custom data and not from transfer learning branch..

Please let me know how to use the coco weights to fine-tune my network with my data. !!!!

Thank you so much for the help !

[glenn-jocher]

@akshu281 to start from the latest pretrained coco weights on yolov3-spp simply run:

python3 train.py --weights yolov3-spp-ultralytics.pt --cfg yolov3-spp.cfg --data yourdatahere

[akshu281]

Thank you Glenn! When I downloaded, the pre-trained weights are ultralytics68.pt which I can run with yolov3-spp.cfg

Just another thing, now using this I can fine-tune on one custom class (for bags) using coco weights, however is it possible to detect one more class (from coco object list) along with this custom class ? How do I train only one custom class and detect another class using the same coco trained as it is good enough ? Is it possible anyways ?

[glenn-jocher]

@akshaygadipatil you can not do what you describe. Any new training will affect existing weights. If you want two classes, you must train on those two classes.

[akshu281]

Understood.. Thank You Glenn I will just use two different models just during inference rather than training them which as you say will affect the existing weights!

[TheodoreKrypton]

In step 5

If you use fewer classes, reduce filters to filters=[4 + 1 + n] * 3, where n is your class count.

I think it should be modified whenever the class count is different from the original.

[glenn-jocher]

@TheodoreKrypton yes good point!

[susanw97]

Hi, I just followed the tutorial for training on custom data and looked at my train_batch0 and test_batch0 images but found that some of the images are cropped and combined (see train_batch0). Is this a normal part of the data augmentation? Does anyone know what is causing this problem?

train_batch0: test_batch0:

Thanks :)

[yangxu351]

It seems like you turn on the "mosaic", It works for me when I set the mosaic=False.

---

From: susanw97 notifications@github.com Sent: Wednesday, March 11, 2020 15:38 To: ultralytics/yolov3 yolov3@noreply.github.com Cc: Yang Xu yang.xu351@duke.edu; Comment comment@noreply.github.com Subject: Re: [ultralytics/yolov3] CUSTOM TRAINING EXAMPLE (#192)

Hi, I just followed the tutorial for training on custom data and looked at my train_batch0 and test_batch0 images but found that some of the images are cropped and combined (see train_batch0). Is this a normal part of the data augmentation? Does anyone know what is causing this problem?

[train_batch0]https://urldefense.com/v3/__https://user-images.githubusercontent.com/28661814/76456669-b7a8b380-63ad-11ea-8184-46cd8a415dd8.png__;!!OToaGQ!9Mq7BSTSICl_BVwZ1o79O7334LaQu0jadXlEQiM2p2ERngS7r8v20FKyebTONE5eDOrUQg$ [test_batch0]https://urldefense.com/v3/__https://user-images.githubusercontent.com/28661814/76456673-b8414a00-63ad-11ea-8f62-4e9485b49a1b.png__;!!OToaGQ!9Mq7BSTSICl_BVwZ1o79O7334LaQu0jadXlEQiM2p2ERngS7r8v20FKyebTONE66PQ-_9w$

Thanks :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/ultralytics/yolov3/issues/192*issuecomment-597829198__;Iw!!OToaGQ!9Mq7BSTSICl_BVwZ1o79O7334LaQu0jadXlEQiM2p2ERngS7r8v20FKyebTONE4QZTf7_A$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AC5CJDNGF732HIPJ4O2QDPDRG7SDRANCNFSM4HEAWLDA__;!!OToaGQ!9Mq7BSTSICl_BVwZ1o79O7334LaQu0jadXlEQiM2p2ERngS7r8v20FKyebTONE4WaC9q-Q$.