Closed glenn-jocher closed 3 years ago
I trained and tested mnist data by using this tutorial. Thank you for guidance.
Hi @glenn-jocher ,
I am trying to train on my custom dataset and I get the following error
Can you please let me know the fix for this error? I see that 'model' class in utils.py does not have an attribute 'hyp'. I followed all the steps outlined in order.
Thanks.
Hi @glenn-jocher ,
I am trying to train on my custom dataset and I get the following error
Can you please let me know the fix for this error? I see that 'model' class in utils.py does not have an attribute 'hyp'. I followed all the steps outlined in order.
Thanks.
I tried on coco_10img.data; I get the same error.
@akshaygadipatil the hyp
attribute contains hyperparameters set in train.py and attached to model
as an easy way to pass the hyperparameters to build_targets()
and compute_losses()
. We just made this change today. Please git pull
to get the absolute latest changes and try again.
Also, what happens if you simply run python3 train.py
?
@akshaygadipatil the example executes correctly on CPU and single GPU. Your issue may be multi-GPU related (you did not specify in your post). If so, git pull
and try again.
python3 train.py --data data/coco_1img.data
Namespace(accumulate=1, backend='nccl', batch_size=16, cfg='cfg/yolov3-spp.cfg', data_cfg='data/coco_1img.data', dist_url='tcp://127.0.0.1:9999', epochs=273, evolve=False, img_size=416, multi_scale=False, nosave=False, notest=False, num_workers=4, rank=0, resume=False, transfer=False, var=0, world_size=1)
Using CPU
layer name gradient parameters shape mu sigma
0 0.conv_0.weight True 864 [32, 3, 3, 3] -0.00339 0.0648
1 0.batch_norm_0.weight True 32 [32] 0.987 1.07
2 0.batch_norm_0.bias True 32 [32] -0.698 2.07
3 1.conv_1.weight True 18432 [64, 32, 3, 3] 0.000298 0.0177
4 1.batch_norm_1.weight True 64 [64] 0.88 0.389
5 1.batch_norm_1.bias True 64 [64] -0.409 1.01
...
223 112.conv_112.weight True 65280 [255, 256, 1, 1] 0.000119 0.0362
224 112.conv_112.bias True 255 [255] -0.000773 0.0356
Model Summary: 225 layers, 6.29987e+07 parameters, 6.29987e+07 gradients
Epoch Batch xy wh conf cls total nTargets time
0/272 0/0 0.192 0.105 15.3 2.36 18 4 5.58
Class Images Targets P R mAP F1
Computing mAP: 100%|████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.90s/it]
all 1 6 0 0 0 0
person 1 3 0 0 0 0
surfboard 1 3 0 0 0 0
Epoch Batch xy wh conf cls total nTargets time
1/272 0/0 0.218 0.0781 15.3 2.36 17.9 5 8.2
Class Images Targets P R mAP F1
Computing mAP: 100%|████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.64s/it]
all 1 6 0 0 0 0
person 1 3 0 0 0 0
surfboard 1 3 0 0 0 0
Epoch Batch xy wh conf cls total nTargets time
2/272 0/0 0.165 0.0669 14.7 2.31 17.2 5 7
Class Images Targets P R mAP F1
Computing mAP: 100%|████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.49s/it]
all 1 6 0 0 0 0
person 1 3 0 0 0 0
surfboard 1 3 0 0 0 0
@glenn-jocher, thanks! Sorry abt not mentioning single/multi gpu usage. I am actually running on a 2-GPU machine.
To be in sync, I tried with the latest changes in the repo. The training has begun. Thank you!
Hi @glenn-jocher ,
Ran into a problem- requesting help:
For some reason, there was a power cut and so the gpu's shut off.
I would like to resume the training process from the latest checkpoint "latest.pt". This was on a multi gpu machine.
I tried changing the weight file in line 87 in train.py
file:
cutoff = load_darknet_weights(model, weights + 'latest.pt')
When I run python3 train.py
, I get an error message:
Can you help me solve this? Thanks!
Hi @glenn-jocher ,
Ran into a problem- requesting help: For some reason, there was a power cut and so the gpu's shut off. I would like to resume the training process from the latest checkpoint "latest.pt". This was on a multi gpu machine. I tried changing the weight file in line 87 in
train.py
file:cutoff = load_darknet_weights(model, weights + 'latest.pt')
When I run
python3 train.py
, I get an error message:Can you help me solve this? Thanks!
Never mind, I should have changed line 67 instead of 87 (in train.py).
BTW, in train.py
, I changed line 122 to- sampler=None
as I was getting an error like as shown below with sampler=sampler
sampler option is mutually exclusive with shuffle
And the error was gone after my fix and the training began ( this was all yesterday). It is not wrong I believe. What do you say?
@akshaygadipatil as the README clearly states https://github.com/ultralytics/yolov3#training
Start Training: python3 train.py
to begin training after downloading COCO data with data/get_coco_dataset.sh
.
Resume Training: python3 train.py --resume
to resume training from weights/latest.pt
.
@glenn-jocher Hi Glen didn't know that this custom training exist. Thanks for the reply earlier, I just abit confuse on how we actually train.
when we run
are we actually training the model to look for the bounding box of a random image(from coco dataset)
because Im confused with step 1 and 2;
where 1 you convert your data into darknet format where it consist of 1.jpg(image) and 1.txt(bounding boxes)
but in 2 we actually train with our coco dataset, not our data set? since the text file is the path of images I guess I just don't get on how to modify #2
@Jriandono you need to create your own *.txt files pointing to your own list of training and testing images. coco_10img.txt
is an example with 10 images in it. Clearly, you make your own if you want to use your own data.
I want to train custom data ,but the following error happened. I think my converted.pt was not correct ,i dont kown how to modify it ,please help me .
Namespace(accumulate=1, backend='nccl', batch_size=1, cfg='cfg/yolov3.cfg', data_cfg='data/coco_10img.data', dist_url='tcp://127.0.0.1:9999', epochs=273, evolve=False, img_size=416, multi_scale=False, nosave=False, notest=False, num_workers=0, rank=0, resume=False, transfer=False, var=0, world_size=1)
Using CUDA device0 _CudaDeviceProperties(name='GeForce GTX 1050', total_memory=2048MB)
Traceback (most recent call last):
File "G:/pycharm/yolo/yolov3-master/train.py", line 309, in <module>
multi_scale=False,
File "G:/pycharm/yolo/yolov3-master/train.py", line 88, in train
chkpt = torch.load(latest, map_location=device) # load checkpoint
File "C:\Users\HP\Anaconda3\envs\wei\lib\site-packages\torch\serialization.py", line 368, in load
return _load(f, map_location, pickle_module)
File "C:\Users\HP\Anaconda3\envs\wei\lib\site-packages\torch\serialization.py", line 532, in _load
magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, '5'.
@you don't need converted.pt to train custom data, you can start training from scratch (i.e. the darknet53 backbone). Just run:
python3 train.py --data data/mycustomfile.data --cfg cfg/mycustomfile.cfg
@you don't need converted.pt to train custom data, you can start training from scratch (i.e. the darknet53 backbone). Just run:
python3 train.py --data data/mycustomfile.data --cfg cfg/mycustomfile.cfg
@you don't need converted.pt to train custom data, you can start training from scratch (i.e. the darknet53 backbone). Just run:
python3 train.py --data data/mycustomfile.data --cfg cfg/mycustomfile.cfg
Thank u so much for your kind reply. if i want to tranfer learning ,how to deal with that question?The converted.pt was created by convert.py in yolov3
@glenn-jocher
First of all, Thank you for creating this repository.
I have followed all the above steps to train the model on my own dataset.
I have 3 classes of samples. so I have modified the filters in *.cfg to filters = 24
but I have one error of
I guess most probably it is due to my image input size. My images are all in the fixed size of 100x100. would you please guide me which part of the code would be affected by this?
@Sam813 this may be related to a recent commit which was fixed. git pull
and try again?
@Sam813 this may be related to a recent commit which was fixed.
git pull
and try again?
Hi @glenn-jocher,
I have tried the new git pull.
After that, I the below error happens in some recently added part of the code.
I have 3 classes, and also modified the data.cfg and *.cfg
@Sam813 your custom data is not configured correctly. If you have 3 classes they should be zero indexed and the class counts in your cfg and .data file should correspond. The error message is saying you are stating 4 classes somewhere and it is not matching up with 3.
train_batch0.jpg
and test_batch0.jpg
for a sanity check of training and testing data.@Sam813 your custom data is not configured correctly. If you have 3 classes they should be zero indexed and the class counts in your cfg and .data file should correspond. The error message is saying you are stating 4 classes somewhere and it is not matching up with 3.
- Your custom data. If your issue is not reproducible with COCO data we can not debug it. Visit our Custom Training Tutorial for exact details on how to format your custom data. Examine
train_batch0.jpg
andtest_batch0.jpg
for a sanity check of training and testing data.
@glenn-jocher Thank you for your help, I found the problem, I had forgotten to set num classes in the cfg file. But now my training results are not making any sense.
all the Precision, recall and F1 are constantly 0. Yet, I can see the confidence is reducing true the training. Do you have any idea whats wrong here?
@Sam813 you are plotting multiple runs sequentially, as results.txt is not erased between runs. If you have zero losses for bounding box regresions, it means you have no bounding boxes to regress, which likely means you have no targets at all, and that the repo can not find your training data.
@Sam813 you are plotting multiple runs sequentially, as results.txt is not erased between runs. If you have zero losses for bounding box regresions, it means you have no bounding boxes to regress, which likely means you have no targets at all, and that the repo can not find your training data.
@glenn-jocher thank you for the help,
If I am not mistaken what you said means my data is not prepared properly?
But I have followed the steps to prepare the data. Moreover, I got this output picture for the test batch which confused me:
Does it have any meaning for you? I guess the bounding boxes have been detected but the image is pure white? could you help to explain it a bit more?
@Sam813 no this is not correct, your data seems to be missing the images. The train_batch0.jpg
file generated when training starts (for correctly prepared data) should look similar to this:
train_batch0.jpg
@glenn-jocher would mind give me an example about "Box coordinates must be in normalized xywh format (from 0 - 1)." I'm a little bit confused about normalized xywh
Dear glenn, I have satellite single channel data and a single class. I already followed the instruction on data preparation and provided bounding boxes; however, I still have two issues. First I want to load images which is different from loading other pictures and I should use gdal for that. then I want to resize them because their size is 512*512 at the moment as well as normalize them and convert them to tensor. The second issue is splitting them into training and validation set. I am following ultralytics code and would like to get some advice on customizing my data in the class LoadImages and class LoadImagesAndLabels(Dataset). Many thanks for any advice.Sanaz
@sanazss this repo handles all image loading, resizing, rescaling and augmentation for a variety of image formats including tif. I recommend you simply follow the example tutorials with your existing data.
I downloade coco data and its structure is like two separate folders for images and labels each contain train and validation set. However I found that you put this path : def coco_class_count(path='coco/images/labels/train2014/'): while there is not any folder named labels in images folder. I am so confused about this structuring. And I think that is one of the reasons I cannot run the code properly on my images.
@sanazss you need to mirror the coco data structure properly to train. The python argument you mention has no place in this discussion, its not called at all during training.
hello, I just got 0% mAP all the time, what should I do to solve it?
@Chida15 start by reproducing the tutorial.
Hi,
I try to apply the tutorial using kitti dataset. I have a GPU so I select device=0 in the train file, but when I am running it is still using the CPU. Anyone had trouble with this?
@GotCstl PyTorch is unable to locate your GPU. You need to install PyTorch and CUDA correctly.
I Manage to launch on GPU. But now I have the following error :
@GotCstl your data is likely not formatted correctly. Start from the tutorial dataset (i.e. coco_64img.data) and go from there.
I am on windows and here is an extract of the text file for train img locations and Data file . Is there any problem?
@glenn-jocher I mange the problem. Image size in config file and train file differs.
Just want to know, how can I get in a file the values for the bounding box coordinates prediction for each test image?
@n0ct4li detect.py lets you save outputs to a text file by setting save_txt=True
:
https://github.com/ultralytics/yolov3/blob/b62dc6f06a4288d759151ea8289da0908f64db2c/detect.py#L9
@glenn-jocher Perfect. And last question, if I want to train on gray-scale images; I just have to set channels = 1 in the config file or is there others things to change? Like maybe number of trainable layers
edit : Got the following error with just changing the channel parameter to 1
@n0ct4li yes for greyscale images set channels=1 in your cfg file.
Beware you may not be able to preload a backbone this way.
Why Am I getting an error?(I put channels=1)
Thank for your sharing.I follow your tutorial and train my custom data on 2 classes, but I got low precison and low mAP.
How to solve this problem?
Hi,
I'm trying to train on my own custom data set for only one class but I get the following error:
I've trained with the same .cfg file on PyTorch before and I'm pretty sure my .data file and my .txt files are correct. But I'm not sure if I need to modify something in train.py to training for one class. I would really appreciate the help. Cheers
@oscarzasa see the single class tutorial in the wiki: https://github.com/ultralytics/yolov3/wiki
@GKDHurryUp read up on vision and ML and follow some of the common options: change your batch size, optimzer, hyperparameters, increase training data, img-size etc.
Hi,
I'm trying to train on my own custom data set for only one class but I get the following error:
I've trained with the same .cfg file on PyTorch before and I'm pretty sure my .data file and my .txt files are correct. But I'm not sure if I need to modify something in train.py to training for one class. I would really appreciate the help. Cheers
Hi,
I'm trying to train on my own custom data set for only one class but I get the following error:
I've trained with the same .cfg file on PyTorch before and I'm pretty sure my .data file and my .txt files are correct. But I'm not sure if I need to modify something in train.py to training for one class. I would really appreciate the help. Cheers
I think you should check the classes and filters number around [yolo] in the .cfg file.There is no need to modify in train.py except some files path.
@GKDHurryUp read up on vision and ML and follow some of the common options: change your batch size, optimzer, hyperparameters, increase training data, img-size etc.
Thank you a lot.I'll try it.
@glenn-jocher thanks mate, I will. Cheers.
@GKDHurryUp I already modified my .cfg file for one class and with the proper number of filters because I previously trained the same file using Darknet. But maybe I missing something else. Thanks tho.
Does anyone known how to normalize the box xywh ? Or for example, an image size is (640,480,3), original box is (38, 75,20, 60)? What is the normalize result??
Here you will find the solution @Hzyu810225
Is it possible initial model weight before training model? i.e. I don't want to training model by load yolo.weight. I want random initial weight.
This guide explains how to train your own custom dataset with YOLOv3.
Before You Start
Clone this repo, download COCO dataset, and install requirements.txt dependencies, including Python>=3.7 and PyTorch>=1.4.
Train On Custom Data
1. Label your data in Darknet format. After using a tool like Labelbox to label your images, you'll need to export your data to darknet format. Your data should follow the example created by
get_coco2017.sh
, with images and labels in separate parallel folders, and one label file per image (if no objects in image, no label file is required). The label file specifications are:class x_center y_center width height
format.x_center
andwidth
by image width, andy_center
andheight
by image height.Each image's label file must be locatable by simply replacing
/images/*.jpg
with/labels/*.txt
in its pathname. An example image and label pair would be:An example label file with 5 persons (all class
0
):*2. Create train and test `.txt
files.** Here we create
data/coco16.txt, which contains the first 16 images of the COCO2017 dataset. We will use this small dataset for both training and testing. Each row contains a path to an image, and remember one label must also exist in a corresponding
/labels` folder for each image containing objects.*3. Create new `.names
file** listing the class names in our dataset. Here we use the existing
data/coco.namesfile. Classes are **zero indexed**, so
personis class
0,
bicycleis class
1`, etc.*4. Create new `.data
file** with your class count (COCO has 80 classes), paths to train and validation datasets (we use the same images twice here, but in practice you'll want to validate your results on a separate set of images), and with the path to your
*.namesfile. Save as
data/coco16.data`.5. Update
yolov3-spp.cfg
(optional). By default each YOLO layer has 255 outputs: 85 values per anchor [4 box coordinates + 1 object confidence + 80 class confidences], times 3 anchors. Update the settings tofilters=[5 + n] * 3
andclasses=n
, wheren
is your class count. This modification should be made in all 3 YOLO layers.6. (OPTIONAL) Update hyperparameters such as LR, LR scheduler, optimizer, augmentation settings, multi_scale settings, etc in
train.py
for your particular task. If in doubt about these settings, we recommend you start with all-default settings before changing anything.7. Train. Run
python3 train.py --cfg yolov3-spp.cfg --data data/coco16.data --nosave
to train using your custom .data and .cfg. By default pretrained--weights yolov3-spp-ultralytics.pt
is used to initialize your model. You can instead train from scratch with--weights ''
, or from any other weights or backbone of your choice, as long as it corresponds to your *.cfg.Visualize Results
Run
from utils import utils; utils.plot_results()
to see your training losses and performance metrics vs epoch. If you don't see acceptable performance, try hyperparameter tuning and re-training. Multipleresults.txt
files are overlaid automatically to compare performance.Here we see training results from
data/coco64.data
starting from scratch, a darknet53 backbone, and ouryolov3-spp-ultralytics.pt
pretrained weights.Run inference with your trained model by copying an image to
![coco_val2014_000000001464](https://user-images.githubusercontent.com/26833433/53104219-42aed580-352f-11e9-9be5-60f84ab05dc1.jpg)
data/samples
folder and runningpython3 detect.py --weights weights/last.pt
Reproduce Our Results
To reproduce this tutorial, simply run the following code. This trains all the various tutorials, saves each results*.txt file separately, and plots them together as
results.png
. It all takes less than 30 minutes on a 2080Ti.Reproduce Our Environment
To access an up-to-date working environment (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled), consider a: