Minimum number of training steps

[Heidisnaps]

Hi.everyone.

I train just 5 class from VOK2012 dataset

training result is ...

step: more than 14000 loss: The value is continuously changed between 2 and 3.

and then I created my-yolo.pb and my-yolo.meta, and prediction like as: flow --pbLoad built_graph/my-yolo.pb --metaLoad built_graph/my-yolo.meta --imgdir sample_img/

but it can't detect anything.

What is the minimum number of steps to detect?

[jubjamie]

Usually until your lost has just converged. You can visualise the training from the summary folder using Tensorboard.

It might not be the number of steps that are the problem. It could be that you're training was never set correctly in the first place, your threshold is too high, different category names etc. Have you managed to do any further troubleshooting?

[Heidisnaps]

@jubjamie

my class:

bicycle bird boat person There were four, not five. However, the number of labels and the number of classes in cfg are the same.

my cfg: `[net] batch=32 subdivisions=8 width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1

learning_rate=0.001 max_batches = 20100 policy=steps steps=-1,100,20000,30000 scales=10,.1,.1

[convolutional] batch_normalize=1 filters=16 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=1

[convolutional] batch_normalize=1 filters=1024 size=3 stride=1 pad=1 activation=leaky

###########

[convolutional] batch_normalize=1 size=3 stride=1 pad=1 filters=1024 activation=leaky

[convolutional] size=1 stride=1 pad=1 filters=45 activation=linear

[region] anchors = 1.08,1.19, 3.42,4.41, 6.63,11.38, 9.42,5.11, 16.62,10.52 bias_match=1 classes=4 coords=4 num=5 softmax=1 jitter=.2 rescore=1

object_scale=5 noobject_scale=1 class_scale=1 coord_scale=1

absolute=1 thresh=.6 random=1`

I chose four classes, but in fact, I just want to find people. Other classes are not important. How do I change the initial value?

[jubjamie]

What initial value?

[Heidisnaps]

@jubjamie

I tried again with 20 classes.

---

~$ flow --train --model cfg/tiny-yolo-voc-new.cfg --dataset "~/VOCdevkit/VOC2007/JPEGImages" --annotation "~/VOCdevkit/VOC2007/Annotations" --gpu 1.0

Parsing cfg/tiny-yolo-voc-new.cfg Loading None ... Finished in 0.00010204315185546875s

Building net ... Source | Train? | Layer description | Output size -------+--------+----------------------------------+--------------- | | input | (?, 416, 416, 3) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 416, 416, 16) Load | Yep! | maxp 2x2p0_2 | (?, 208, 208, 16) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 208, 208, 32) Load | Yep! | maxp 2x2p0_2 | (?, 104, 104, 32) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 104, 104, 64) Load | Yep! | maxp 2x2p0_2 | (?, 52, 52, 64) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 52, 52, 128) Load | Yep! | maxp 2x2p0_2 | (?, 26, 26, 128) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 26, 26, 256) Load | Yep! | maxp 2x2p0_2 | (?, 13, 13, 256) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 512) Load | Yep! | maxp 2x2p0_1 | (?, 13, 13, 512) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 1x1p0_1 linear | (?, 13, 13, 125) -------+--------+----------------------------------+--------------- GPU mode with 1.0 usage

cfg/tiny-yolo-voc-new.cfg loss hyper-parameters: H = 13 W = 13 box = 5 classes = 20 scales = [1.0, 5.0, 1.0, 1.0]

Parsing for ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'] [====================>]100% 007614.xml Statistics: bird: 599 train: 328 diningtable: 310 chair: 1432 bus: 272 dog: 538 cat: 389 bottle: 634 motorbike: 390 tvmonitor: 367 person: 5447 pottedplant: 625 sheep: 353 bicycle: 418 car: 1644 horse: 406 aeroplane: 331 sofa: 425 cow: 356 boat: 398 Dataset size: 5011 Dataset of 5011 instance(s) Training statistics: Learning rate : 1e-05 Batch size : 16 Epoch number : 1000 Backup every : 2000

step 1 - loss 109.52924346923828 - moving ave loss 109.52924346923828 step 2 - loss 109.86571502685547 - moving ave loss 109.56289062500001 step 3 - loss 110.08113098144531 - moving ave loss 109.61471466064454 . . . step 12492 - loss 10.850701332092285 - moving ave loss 8.618697419224091 step 12493 - loss 6.189124584197998 - moving ave loss 8.375740135721482 step 12494 - loss 9.410148620605469 - moving ave loss 8.479180984209881 step 12495 - loss 7.525200843811035 - moving ave loss 8.383782970169996

~$ flow --model cfg/tiny-yolo-voc-new.cfg --load -1 --savepb
~$ flow --pbLoad built_graph/tiny-yolo-voc-new.pb --metaLoad built_graph/tiny-yolo-voc-new.meta --imgdir sample_img/ --json

But this also does not detect anything.

[jubjamie]

does it detect before you translate to pb files? Check to see if it's a translation error. I also don't see where you are saving your weights? What if you use transfer learning?

[Gowan1998]

Hey Heidisnaps,

i think you have the same problem as i have. The problem is, that after a training with more than 14000 stages/steps savepb doesn't work anymore! If you build the savepb file and want to detect something with it in Android (or use : flow --pbLoad built_graph/yolo-new.pb --metaLoad built_graph/yolo-new.meta --imgdir sample_img/) it doesnt works :(

I think there is a Bug, because the weights works right. For example you can try: ./flow --imgdir sample_img/ --model cfg/yolo-new.cfg --load 1500 and it works!

And the problem appears after 14000 training steps! I try a lot of combinations and train more than 600000 steps and it doesnt works! But with less than 14000 steps you didnt have any problems!

[Heidisnaps]

@jubjamie

Translate to pb files is no error ~$ flow --model cfg/tiny-yolo-voc-new.cfg --load 153 --savepb

Parsing cfg/tiny-yolo-voc-new.cfg Loading None ... Finished in 0.00011944770812988281s

Building net ... Source | Train? | Layer description | Output size -------+--------+----------------------------------+--------------- | | input | (?, 416, 416, 3) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 416, 416, 16) Load | Yep! | maxp 2x2p0_2 | (?, 208, 208, 16) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 208, 208, 32) Load | Yep! | maxp 2x2p0_2 | (?, 104, 104, 32) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 104, 104, 64) Load | Yep! | maxp 2x2p0_2 | (?, 52, 52, 64) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 52, 52, 128) Load | Yep! | maxp 2x2p0_2 | (?, 26, 26, 128) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 26, 26, 256) Load | Yep! | maxp 2x2p0_2 | (?, 13, 13, 256) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 512) Load | Yep! | maxp 2x2p0_1 | (?, 13, 13, 512) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 1x1p0_1 linear | (?, 13, 13, 125) -------+--------+----------------------------------+--------------- Running entirely on CPU Loading from ./ckpt/tiny-yolo-voc-new-153 Finished in 7.258143663406372s Rebuild a constant version ... Done

[Heidisnaps]

@Savash2016 I tried under 14000. like as: ~$ flow --model cfg/tiny-yolo-voc-new.cfg --load 12000 --savepb

it is work!thanks!! But the accuracy is too low. How did you improve the accuracy?

[Gowan1998]

Hey Heidisnaps,

its a bug i think :( The only way to get a better accuracy is to train more than 300000 steps (depends on your data). But the problem is, if you do this your pb file does not work anymore.

I hope the author will fix it soon. Maybe you can try also send him a Mail and ask him for fixing :/ I think this is the only way.

[junxuezheng]

@Heidisnaps
Hey Heidisnaps,can you send to me the code about the accuracy. thanks,my gmail junxuezheng@gmail.com

[junxuezheng]

@Savash2016 hey Savash2016 ;

the cfg have the max_batches ,in yolo-voc max_batches is 45000,in tiny-yolo-voc max_batches is 40100(voc 2007). i think the max_batches is the max_step. i want to get a better accuracy,does it need to train more than 300000 steps? and,when it run above 40000 steps ,the loss does not descend any more.( the loss is between 4.0-5.0) thanks ,my gmai is junxuezheng@gmai.com

[Heidisnaps]

@junxuezheng Sorry, but I do not have any code at this time.