search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.77k stars 7.96k forks source link

Got an exception when calculating map #7714

Open FortegaITSolutions opened 3 years ago

FortegaITSolutions commented 3 years ago

Calculating map works correct: when I run the following, it works: darknet.exe detector map obj.data piecedetection.cfg backup/piecedetection_last.weights

However, when I run the training with '-map', it fails while calucating the map for the first time

info darknet.exe detector train obj.data piecedetection.cfg backup/piecedetection_last.weights -map CUDA-version: 11030 (11030), cuDNN: 8.1.1, GPU count: 1 OpenCV version: 4.5.2 Prepare additional network for mAP calculation... 0 : compute_capability = 860, cudnn_half = 0, GPU: NVIDIA GeForce RTX 3080 net.optimized_memory = 0 mini_batch = 1, batch = 16, time_steps = 1, train = 0 layer filters size/strd(dil) input output 0 Create CUDA-stream - 0 Create cudnn-handle 0

error (next mAP calculation at 1089 iterations) 1089: 1.256688, 0.946720 avg loss, 0.001000 rate, 2.553000 seconds, 34848 images, 19.219721 hours left Resizing to initial size: 480 x 480 try to allocate additional workspace_size = 88.85 MB CUDA allocate done!

calculation mAP (mean average precision)... Detection layer: 139 - type = 28 Detection layer: 150 - type = 28 Detection layer: 161 - type = 28 4 cuDNN status Error in: file: C:\Program Files\darknet-master\src\convolutional_kernels.cu : forward_convolutional_layer_gpu() : line: 553 : build time: May 12 2021 - 13:22:54

cuDNN Error: CUDNN_STATUS_BAD_PARAM darknet.exe detector train obj.data piecedetection.cfg backup/piecedetection_last.weights -map cuDNN Error: CUDNN_STATUS_BAD_PARAM: No error

AimplainLeo commented 3 years ago

May be your wrong valid dataset, no exist file or bad image (gif ... ).

FortegaITSolutions commented 3 years ago

May be your wrong valid dataset, no exist file or bad image (gif ... ).

I assume this is not the case, as the 'map' call did work correctly:

darknet.exe detector map obj.data piecedetection.cfg backup/piecedetection_last.weights

Anyway, even if this is the case, is there a way of getting a better error message?

AimplainLeo commented 3 years ago

Simple way, i write a tool for checking duplicate images and extension, remove or notice for editting then -> re-run maker tool for generating annotations again. Anyway, even if this is the case, is there a way of getting a better error message? I dont know 👯‍♂️

elias-work commented 3 years ago

I recall having this issue on a GPU that didn't support cudnn_half. Since you have cudnn_half disabled though your GPU supports it, perhaps it could be the same issue as me but different direct cause? You will get a speed boost using it anyway but I'd be interested to see if you still have the issue with it on.

FortegaITSolutions commented 3 years ago

Enabling cudnn_half didn't solve the issue. (But thanks for pointing that out, the speed boost is indeed significant!) I recreated the train.txt and valid.txt lists based on the images which are present, and that solved the problem for me.