Open bhavyabpk opened 4 years ago
It seems something wrong with your validation dataset.
Show obj.dat file Check bad.list and bad_label.list files Are there any other error messages? What params did you set in the Makefile? Set valid=train.txt in obj.data file
This is the plot i am getting
As you can see inspite of -map flag, only loss is getting plotted.
Initially it says in log that map will be calculated at 400 iteration but after 399 iteration the process hangs and in task manager its written not responsing in front of darknet task. i waited for hours but it just remained hanged. So i terminated the process by pressing ctrl+c and this was the output -
4^ZCUDA Error Prev: unspecified launch failure: No error Assertion failed: 0, file c:\users\bhavya\python\darknet-master\src\utils.c, line 297
this is how i am training -
darknet.exe detector train tinyyolov3\obj.data tinyyolov3\yolov3-tiny_obj.cfg tinyyolov3\yolov3-tiny.conv.15 > tinyyolov3\train.log -map
this is the cfg file
`[net]
Testing
batch=1
subdivisions=1
Training
batch=64 subdivisions=16 width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1
learning_rate=0.001 burn_in=400 max_batches = 7200 policy=steps steps=4800,5400 scales=.1,.1
[convolutional] batch_normalize=1 filters=16 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=1
[convolutional] batch_normalize=1 filters=1024 size=3 stride=1 pad=1 activation=leaky
###########
[convolutional] batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky
[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky
[convolutional] size=1 stride=1 pad=1 filters=24 activation=linear
[yolo] mask = 3,4,5 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1
[route] layers = -4
[convolutional] batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky
[upsample] stride=2
[route] layers = -1, 8
[convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky
[convolutional] size=1 stride=1 pad=1 filters=24 activation=linear
[yolo] mask = 0,1,2 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1 `
@bhavyabpk hi did you solve this problem, i'm also facing this issue now...
This is the plot i am getting
As you can see inspite of -map flag, only loss is getting plotted. Initially it says in log that map will be calculated at 400 iteration but after 399 iteration the process hangs and in task manager its written not responsing in front of darknet task. i waited for hours but it just remained hanged. So i terminated the process by pressing ctrl+c and this was the output -
4^ZCUDA Error Prev: unspecified launch failure: No error Assertion failed: 0, file c:\users\bhavya\python\darknet-master\src\utils.c, line 297
this is how i am training -darknet.exe detector train tinyyolov3\obj.data tinyyolov3\yolov3-tiny_obj.cfg tinyyolov3\yolov3-tiny.conv.15 > tinyyolov3\train.log -map
this is the cfg file `[net]Testing
batch=1
subdivisions=1
Training
batch=64 subdivisions=16 width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1 learning_rate=0.001 burn_in=400 max_batches = 7200 policy=steps steps=4800,5400 scales=.1,.1 [convolutional] batch_normalize=1 filters=16 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky [maxpool] size=2 stride=1 [convolutional] batch_normalize=1 filters=1024 size=3 stride=1 pad=1 activation=leaky ########### [convolutional] batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky [convolutional] size=1 stride=1 pad=1 filters=24 activation=linear [yolo] mask = 3,4,5 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1 [route] layers = -4 [convolutional] batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky [upsample] stride=2 [route] layers = -1, 8 [convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky [convolutional] size=1 stride=1 pad=1 filters=24 activation=linear [yolo] mask = 0,1,2 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1 `
@bhavyabpk hi did you solve this problem, i'm also facing this issue now...
Yes, there were some images which were not of proper format. So if you can make sure or convert all the images properly to a common format, this issue may get resolved. Even if the extension is same but there may be some images which are actually png but the extension is jpeg. in such cases error may come. So try fixing this.
This is the plot i am getting
As you can see inspite of -map flag, only loss is getting plotted.
Initially it says in log that map will be calculated at 400 iteration but after 399 iteration the process hangs and in task manager its written not responsing in front of darknet task. i waited for hours but it just remained hanged. So i terminated the process by pressing ctrl+c and this was the output -
4^ZCUDA Error Prev: unspecified launch failure: No error Assertion failed: 0, file c:\users\bhavya\python\darknet-master\src\utils.c, line 297
this is how i am training -
darknet.exe detector train tinyyolov3\obj.data tinyyolov3\yolov3-tiny_obj.cfg tinyyolov3\yolov3-tiny.conv.15 > tinyyolov3\train.log -map
this is the cfg file
`[net]
Testing
batch=1
subdivisions=1
Training
batch=64 subdivisions=16 width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1
learning_rate=0.001 burn_in=400 max_batches = 7200 policy=steps steps=4800,5400 scales=.1,.1
[convolutional] batch_normalize=1 filters=16 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=2
[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky
[maxpool] size=2 stride=1
[convolutional] batch_normalize=1 filters=1024 size=3 stride=1 pad=1 activation=leaky
###########
[convolutional] batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky
[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky
[convolutional] size=1 stride=1 pad=1 filters=24 activation=linear
[yolo] mask = 3,4,5 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1
[route] layers = -4
[convolutional] batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky
[upsample] stride=2
[route] layers = -1, 8
[convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky
[convolutional] size=1 stride=1 pad=1 filters=24 activation=linear
[yolo] mask = 0,1,2 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=3 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1 `