Open mayur-ag opened 3 years ago
It is stuck here at this point. But when I use:
It looks like you specified wrong cfg or weights file.
@AlexeyAB Sorry, I've edited the post. I have passed the right paths, but edited it here to hide local paths.
The same behaviour is also observed in the darknet detector train
command. During the mAP calculation it's extremely slow. Takes like 30 secs per image.
The same behaviour is also observed in the darknet detector train
command. During the mAP calculation it's extremely slow. Takes like 30 secs per image.
.01
and recompile: https://github.com/AlexeyAB/darknet/blob/103d301ccbc19e47e002005bdfdbaf07a92cd880/src/detector.c#L995Isn't it same as passing -thresh
while running the command? I'll try recompile and test. Here's the cfg file, it's just yolov4-tiny
[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=1
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1
learning_rate=0.00261
burn_in=1000
max_batches = 20000
policy=steps
steps=15000,17500
scales=.1,.1
[convolutional]
batch_normalize=1
filters=32
size=3
stride=2
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky
[route]
layers=-1
groups=2
group_id=1
[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky
[route]
layers = -1,-2
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky
[route]
layers = -6,-1
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[route]
layers=-1
groups=2
group_id=1
[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky
[route]
layers = -1,-2
[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky
[route]
layers = -6,-1
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky
[route]
layers=-1
groups=2
group_id=1
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[route]
layers = -1,-2
[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky
[route]
layers = -6,-1
[maxpool]
size=2
stride=2
[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky
##################################
[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky
[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky
[convolutional]
size=1
stride=1
pad=1
filters=117
activation=linear
[yolo]
mask = 6,7,8
anchors = 34, 91, 26,148, 37,148, 55,111, 32,237, 49,175, 267, 38, 45,283, 84,158
classes=34
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6
[route]
layers = -4
[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky
[upsample]
stride=2
[route]
layers = -1, 23
[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky
[convolutional]
size=1
stride=1
pad=1
filters=117
activation=linear
[yolo]
mask = 3,4,5
anchors = 34, 91, 26,148, 37,148, 55,111, 32,237, 49,175, 267, 38, 45,283, 84,158
classes=34
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6
[route]
layers = -3
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky
[upsample]
stride=2
[route]
layers = -1, 15
[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky
[convolutional]
size=1
stride=1
pad=1
filters=117
activation=linear
[yolo]
mask = 0,1,2
anchors = 34, 91, 26,148, 37,148, 55,111, 32,237, 49,175, 267, 38, 45,283, 84,158
classes=34
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6
I have recompiled by replacing the threshold variable in detector.c
, but the issue still persists.
Try to set this value to .2
and recompile: https://github.com/AlexeyAB/darknet/blob/103d301ccbc19e47e002005bdfdbaf07a92cd880/src/detector.c#L995
If it will not help, then the issue is that the cfg file does not match the weight file, or the weights are poorly trained.
i had that issue with 3090 when opencv wasnt compiled correctly.
I have OpenCV with CUDA support compiled from source. Is this combination known to have problems? @AlexeyAB @haviduck
Another observation is that: this problem occurs only on CERTAIN types of models. For instance, the one presented above. I trained another model with a different config (YOLOv4x) and that worked flawlessly. Also, I think this issue has nothing to do with good/bad model convergence, as the problematic model freezes even after it has converged.
hmm, well it certainly can be depending on the type of layers in your cfg. but if i recall correctly, cuda 11 only works with opencv 4.5 for the 3x series. thats what im running anyway. i spent alot of time compiling opencv correctly, then made sure the proper dlls was in place. i checked that by systematically going through each dll with dumpbin /DEPENDENTS file.dll. it gives you a list over required dlls. if some of them are outside your env paths you need to place them in the same dir as your script. you also have to recompile the console dll and nogpu dll found in the!+ /build directory and ensure you replace any line where 10.0 or 10.1 is hardcoded.
but i can only talk from my own setup and experience. opencv_world, opencv_videoio and ffmpeg along with others needs to be in your paths. oh and remember to add the path to opencv contrib bin when you compile opencv.
this guide is pretty awesome https://jamesbowley.co.uk/accelerate-opencv-4-5-0-on-windows-build-with-cuda-and-python-bindings/
@mayur-ag @AlexeyAB I have the same issue with yolov4-csp network. Training speed is normal, yet map calculation is extremely slow.
Yolov4-tiny training with the same data has no such issue. So I doubt the problem is with the darknet build.
@kadirbeytorun yeah I faced the same issue, the problem is only with "certain" types of configurations. For instance, dataset 1 - yolov4-tiny has the problem, but dataset-2 yolov4-tiny wouldn't have it. I tried to debug the source code but eventually gave up.
What I settled for is to NOT use the -map
option while training. Once the model is trained, I just use darknet map
command the check the map. Only downside is, you won't have the training mAP graph.
im doing something different I trained the model with 1 cfg , and i got the weight of that model .so i was doing testing on same data but only weight is different, im getting no results.
I have Quetion, my approach is wrong? Do we need to find MAP on the same data, same weight & cfg??
I have the same problem, training is normal, but the -map calculation is extremely slow.
During training mAP calculation, and even while using the
darknet detector map
command, the calculation is painfully slow. It takes several seconds to read a single image. I tried editing mytest.txt
to have only 2 images, and it's again stuck. Following is the snapshot:It is stuck here at this point. But when I use:
It runs very fast:
It means it's stuck while calculating mAP. I've tried disabling OPENMP, AVX, but still the same issue.